2024-04-01T04:38:24.120 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-01T04:38:24.121 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi/7632907 --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} --owner scheduled_teuthology@teuthology --name teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi/7632907/orig.config.yaml 2024-04-01T04:38:24.143 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T04:38:24.222 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi/7632907 branch: squid description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: ceph-qa@ceph.com first_in_suite: false job_id: '7632907' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1304 sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 sleep_before_teardown: 0 subset: 13190/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDVqi0A7C6HAQGxP/GypzQQl4gTlLvjVme2mh56L5LfdYSfdgC5RR/+RGzME5WAzu9ZQnaA3Xr/nrpkBwgJBuws= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-31_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-01T04:38:24.222 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa; will attempt to use it 2024-04-01T04:38:24.223 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks 2024-04-01T04:38:24.223 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-01T04:38:24.225 INFO:teuthology.task.internal:Checking packages... 2024-04-01T04:38:24.247 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '384f31f3d3aee7f8ba3f60846b8844c049165583' 2024-04-01T04:38:24.248 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-01T04:38:24.248 INFO:teuthology.packaging:ref: None 2024-04-01T04:38:24.248 INFO:teuthology.packaging:tag: None 2024-04-01T04:38:24.248 INFO:teuthology.packaging:branch: squid 2024-04-01T04:38:24.248 INFO:teuthology.packaging:sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T04:38:24.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-01T04:38:24.506 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1977.g384f31f3 2024-04-01T04:38:24.507 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-01T04:38:24.514 INFO:teuthology.task.internal:no buildpackages task found 2024-04-01T04:38:24.514 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-01T04:38:24.562 INFO:teuthology.task.internal:Saving configuration 2024-04-01T04:38:24.572 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-01T04:38:24.587 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T04:38:24.613 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi/7632907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 04:33:16.457646', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDVqi0A7C6HAQGxP/GypzQQl4gTlLvjVme2mh56L5LfdYSfdgC5RR/+RGzME5WAzu9ZQnaA3Xr/nrpkBwgJBuws='} 2024-04-01T04:38:24.613 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-01T04:38:24.619 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-01T04:38:24.619 INFO:teuthology.run_tasks:Running task console_log... 2024-04-01T04:38:24.681 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc218665e50>, signals=[15]) 2024-04-01T04:38:24.682 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-01T04:38:24.688 INFO:teuthology.task.internal:Opening connections... 2024-04-01T04:38:24.688 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-01T04:38:24.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T04:38:24.796 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-01T04:38:24.802 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-01T04:38:24.872 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-01T04:38:24.873 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-01T04:38:24.928 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-04-01T04:38:24.928 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-04-01T04:38:24.928 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T04:38:24.929 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T04:38:24.930 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-01T04:38:24.961 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-01T04:38:24.971 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-01T04:38:24.987 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-01T04:38:24.987 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-01T04:38:25.003 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-01T04:38:25.010 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-01T04:38:25.010 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-01T04:38:25.067 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-01T04:38:25.101 INFO:teuthology.run_tasks:Running task kernel... 2024-04-01T04:38:25.114 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-01T04:38:25.114 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-01T04:38:25.115 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T04:38:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T04:38:25.132 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-01T04:38:25.187 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2024-04-01T04:38:25.188 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-361.el9.x86_64 2024-04-01T04:38:25.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-04-01T04:38:27.287 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-01T04:38:28.713 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-04-01T04:38:29.762 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-01T04:38:31.949 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-01T04:38:36.755 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.1 MB 00:01 2024-04-01T04:38:38.607 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-01T04:38:40.651 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-01T04:38:45.841 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-04-01T04:38:48.122 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T04:38:48.123 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T04:38:48.219 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:38:48.228 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:38:48.229 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2024-04-01T04:38:48.230 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:38:48.230 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 76 M 2024-04-01T04:38:48.230 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 123 M 2024-04-01T04:38:48.230 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:38:48.375 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-01T04:38:49.009 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-04-01T04:38:49.476 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-04-01T04:38:49.851 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-01T04:38:49.853 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:38:49.854 INFO:teuthology.orchestra.run.smithi067.stdout:Total 47 MB/s | 76 MB 00:01 2024-04-01T04:38:49.854 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:38:50.249 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:38:50.249 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:38:51.123 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:38:51.123 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:38:53.535 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:38:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T04:38:54.259 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T04:38:57.136 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T04:38:57.277 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T04:39:02.357 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T04:39:02.401 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T04:39:07.498 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T04:39:37.979 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T04:39:37.979 INFO:teuthology.orchestra.run.smithi067.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T04:39:37.979 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:39:38.022 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T04:39:38.440 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T04:39:38.440 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T04:39:38.440 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T04:39:38.440 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:39:38.866 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:39:39.182 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T04:39:39.296 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T04:39:39.296 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T04:39:39.296 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T04:39:39.296 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T04:39:39.296 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T04:39:39.297 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-01T04:39:39.297 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T04:39:39.297 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T04:39:39.379 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T04:39:39.380 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T04:39:39.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2024-04-01T04:39:39.426 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2024-04-01T04:39:39.426 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2024-04-01T04:39:39.426 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2024-04-01T04:39:39.426 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2024-04-01T04:39:39.426 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2024-04-01T04:39:39.426 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T04:39:39.427 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T04:39:39.428 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2024-04-01T04:39:39.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:39:39.428 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T04:39:39.428 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T04:39:39.429 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-434.el9.x86_64 2024-04-01T04:39:39.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T04:39:39.989 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-01T04:39:41.341 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T04:39:41.376 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-01T04:39:41.379 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-04-01T04:39:41.394 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.l3keZ3Xj0m 2024-04-01T04:39:41.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.l3keZ3Xj0m 2024-04-01T04:39:41.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.l3keZ3Xj0m 2024-04-01T04:39:41.587 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.l3keZ3Xj0m is 6KB 2024-04-01T04:39:41.636 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.l3keZ3Xj0m 2024-04-01T04:39:41.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T04:39:41.719 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T04:39:41.719 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T04:39:41.719 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T04:39:41.719 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T04:39:41.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T04:39:41.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-04-01T04:40:11.934 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-01T04:40:11.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:40:11.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T04:40:30.495 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-01T04:40:39.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:40:39.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T04:40:42.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-01T04:40:54.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:40:54.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T04:41:12.991 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-01T04:41:28.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:41:28.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T04:41:28.364 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-01T04:41:28.741 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:41:28.742 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-01T04:41:28.742 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-01T04:41:28.756 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-434.el9.x86_64 2024-04-01T04:41:28.756 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T04:41:28.756 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T04:41:28.756 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T04:41:29.757 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-01T04:41:29.758 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T04:41:29.834 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-04-01T04:41:29.868 DEBUG:teuthology.parallel:result is None 2024-04-01T04:41:29.869 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-01T04:41:29.877 INFO:teuthology.task.internal:Creating test directory... 2024-04-01T04:41:29.877 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T04:41:29.891 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-01T04:41:29.908 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-01T04:41:29.925 INFO:teuthology.task.internal:Creating archive directory... 2024-04-01T04:41:29.926 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T04:41:29.956 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-01T04:41:29.963 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-01T04:41:29.963 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T04:41:30.024 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:41:30.038 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:41:30.039 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-01T04:41:30.047 INFO:teuthology.task.internal:Configuring sudo... 2024-04-01T04:41:30.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T04:41:30.106 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-01T04:41:30.115 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-01T04:41:30.115 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T04:41:30.160 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T04:41:30.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T04:41:30.285 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T04:41:30.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T04:41:30.407 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:41:30.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T04:41:30.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-01T04:41:30.557 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T04:41:31.002 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-01T04:41:31.010 INFO:teuthology.task.internal:Starting timer... 2024-04-01T04:41:31.010 INFO:teuthology.run_tasks:Running task pcp... 2024-04-01T04:41:31.037 INFO:teuthology.run_tasks:Running task selinux... 2024-04-01T04:41:31.052 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-01T04:41:31.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-04-01T04:41:31.125 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-04-01T04:41:31.127 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-01T04:41:31.127 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-04-01T04:41:31.163 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-04-01T04:41:31.163 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2024-04-01T04:41:31.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T04:41:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T04:41:31.229 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-01T04:41:31.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-04-01T04:41:31.332 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-01T04:41:31.341 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-01T04:41:31.356 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-01T04:41:31.357 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-01T04:45:54.406 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2024-04-01T04:45:54.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:45:54.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T04:45:54.498 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-01T04:45:54.577 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-01T04:45:54.577 INFO:teuthology.run_tasks:Running task clock... 2024-04-01T04:45:54.600 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-01T04:45:54.600 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T04:45:54.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T04:45:54.649 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T04:45:54.666 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T04:45:54.695 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-04-01T04:45:54.709 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-04-01T04:45:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T04:45:54.742 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T04:45:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-01T04:45:54.797 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T04:45:54.797 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-01T04:45:54.797 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T04:45:54.797 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T04:45:54.797 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T04:45:54.797 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T04:45:54.798 INFO:teuthology.run_tasks:Running task install... 2024-04-01T04:45:54.806 DEBUG:teuthology.task.install:project ceph 2024-04-01T04:45:54.806 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583'}} 2024-04-01T04:45:54.806 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583'} 2024-04-01T04:45:54.806 INFO:teuthology.task.install:Using flavor: default 2024-04-01T04:45:54.812 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-01T04:45:54.812 INFO:teuthology.task.install:extra packages: [] 2024-04-01T04:45:54.812 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-04-01T04:45:54.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T04:45:55.118 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-04-01T04:45:55.118 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-04-01T04:45:55.369 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T04:45:55.369 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:45:55.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T04:45:55.399 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-01T04:45:55.399 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T04:45:55.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T04:45:55.551 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T04:45:55.618 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-04-01T04:45:55.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-04-01T04:45:55.878 INFO:teuthology.orchestra.run.smithi067.stdout:74 files removed 2024-04-01T04:45:55.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2024-04-01T04:45:56.832 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 134 kB/s | 83 kB 00:00 2024-04-01T04:45:57.299 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-04-01T04:45:57.766 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-04-01T04:45:58.435 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-01T04:46:00.044 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-04-01T04:46:01.853 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-01T04:46:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-01T04:46:09.353 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-04-01T04:46:12.087 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 16 kB 00:01 2024-04-01T04:46:15.514 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-04-01T04:46:20.655 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-04-01T04:46:23.349 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-04-01T04:46:23.357 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T04:46:23.358 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-04-01T04:46:23.359 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:Install 30 Packages 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2024-04-01T04:46:23.360 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:23.361 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 59 M 2024-04-01T04:46:23.361 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:46:25.173 INFO:teuthology.orchestra.run.smithi067.stdout:(1/32): ceph-base-19.0.0-1977.g384f31f3.el9.x86 3.0 MB/s | 5.3 MB 00:01 2024-04-01T04:46:25.265 INFO:teuthology.orchestra.run.smithi067.stdout:(2/32): ceph-selinux-19.0.0-1977.g384f31f3.el9. 274 kB/s | 25 kB 00:00 2024-04-01T04:46:25.440 INFO:teuthology.orchestra.run.smithi067.stdout:(3/32): ceph-radosgw-19.0.0-1977.g384f31f3.el9. 4.8 MB/s | 9.8 MB 00:02 2024-04-01T04:46:25.507 INFO:teuthology.orchestra.run.smithi067.stdout:(4/32): libradosstriper1-19.0.0-1977.g384f31f3. 7.4 MB/s | 501 kB 00:00 2024-04-01T04:46:25.532 INFO:teuthology.orchestra.run.smithi067.stdout:(5/32): libcephfs2-19.0.0-1977.g384f31f3.el9.x8 2.7 MB/s | 746 kB 00:00 2024-04-01T04:46:25.599 INFO:teuthology.orchestra.run.smithi067.stdout:(6/32): python3-ceph-argparse-19.0.0-1977.g384f 675 kB/s | 45 kB 00:00 2024-04-01T04:46:25.675 INFO:teuthology.orchestra.run.smithi067.stdout:(7/32): python3-ceph-common-19.0.0-1977.g384f31 1.7 MB/s | 131 kB 00:00 2024-04-01T04:46:25.750 INFO:teuthology.orchestra.run.smithi067.stdout:(8/32): python3-cephfs-19.0.0-1977.g384f31f3.el 2.1 MB/s | 161 kB 00:00 2024-04-01T04:46:25.842 INFO:teuthology.orchestra.run.smithi067.stdout:(9/32): python3-rados-19.0.0-1977.g384f31f3.el9 3.4 MB/s | 320 kB 00:00 2024-04-01T04:46:25.934 INFO:teuthology.orchestra.run.smithi067.stdout:(10/32): python3-rbd-19.0.0-1977.g384f31f3.el9. 3.2 MB/s | 298 kB 00:00 2024-04-01T04:46:26.201 INFO:teuthology.orchestra.run.smithi067.stdout:(11/32): ceph-common-19.0.0-1977.g384f31f3.el9. 7.4 MB/s | 21 MB 00:02 2024-04-01T04:46:26.226 INFO:teuthology.orchestra.run.smithi067.stdout:(12/32): python3-rgw-19.0.0-1977.g384f31f3.el9. 340 kB/s | 99 kB 00:00 2024-04-01T04:46:26.418 INFO:teuthology.orchestra.run.smithi067.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 488 kB/s | 106 kB 00:00 2024-04-01T04:46:26.469 INFO:teuthology.orchestra.run.smithi067.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 734 kB/s | 159 kB 00:00 2024-04-01T04:46:26.494 INFO:teuthology.orchestra.run.smithi067.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-04-01T04:46:26.627 INFO:teuthology.orchestra.run.smithi067.stdout:(16/32): librgw2-19.0.0-1977.g384f31f3.el9.x86_ 4.6 MB/s | 5.1 MB 00:01 2024-04-01T04:46:26.652 INFO:teuthology.orchestra.run.smithi067.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-04-01T04:46:26.678 INFO:teuthology.orchestra.run.smithi067.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 282 kB 00:00 2024-04-01T04:46:26.703 INFO:teuthology.orchestra.run.smithi067.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 554 kB/s | 41 kB 00:00 2024-04-01T04:46:26.728 INFO:teuthology.orchestra.run.smithi067.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 699 kB/s | 35 kB 00:00 2024-04-01T04:46:26.754 INFO:teuthology.orchestra.run.smithi067.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-04-01T04:46:26.779 INFO:teuthology.orchestra.run.smithi067.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 612 kB/s | 46 kB 00:00 2024-04-01T04:46:26.854 INFO:teuthology.orchestra.run.smithi067.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-04-01T04:46:26.904 INFO:teuthology.orchestra.run.smithi067.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-04-01T04:46:26.929 INFO:teuthology.orchestra.run.smithi067.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-04-01T04:46:26.955 INFO:teuthology.orchestra.run.smithi067.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-01T04:46:27.039 INFO:teuthology.orchestra.run.smithi067.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-01T04:46:27.064 INFO:teuthology.orchestra.run.smithi067.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-01T04:46:27.106 INFO:teuthology.orchestra.run.smithi067.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-04-01T04:46:27.147 INFO:teuthology.orchestra.run.smithi067.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-01T04:46:27.398 INFO:teuthology.orchestra.run.smithi067.stdout:(31/32): librbd1-19.0.0-1977.g384f31f3.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-04-01T04:46:27.723 INFO:teuthology.orchestra.run.smithi067.stdout:(32/32): librados2-19.0.0-1977.g384f31f3.el9.x8 5.1 MB/s | 3.4 MB 00:00 2024-04-01T04:46:27.727 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:46:27.727 INFO:teuthology.orchestra.run.smithi067.stdout:Total 13 MB/s | 59 MB 00:04 2024-04-01T04:46:27.728 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:46:27.770 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:46:27.770 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:46:28.268 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:46:28.268 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:46:29.791 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:46:29.879 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-01T04:46:29.920 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-01T04:46:29.945 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-01T04:46:29.976 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-01T04:46:29.992 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-01T04:46:30.176 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-01T04:46:30.232 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-01T04:46:30.604 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-01T04:46:30.638 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 7/34 2024-04-01T04:46:30.714 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 7/34 2024-04-01T04:46:30.805 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 8/34 2024-04-01T04:46:30.840 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 9/34 2024-04-01T04:46:30.894 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 9/34 2024-04-01T04:46:30.923 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-01T04:46:30.967 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9. 11/34 2024-04-01T04:46:31.045 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 12/34 2024-04-01T04:46:31.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_6 13/34 2024-04-01T04:46:31.148 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_6 13/34 2024-04-01T04:46:31.182 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-01T04:46:31.294 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-01T04:46:31.334 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-01T04:46:31.374 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-01T04:46:31.407 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-01T04:46:31.436 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-01T04:46:31.468 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-01T04:46:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-01T04:46:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 22/34 2024-04-01T04:46:31.914 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 22/34 2024-04-01T04:46:31.944 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 23/34 2024-04-01T04:46:31.989 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-01T04:46:32.467 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-01T04:46:32.501 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 26/34 2024-04-01T04:46:32.581 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 26/34 2024-04-01T04:46:32.652 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 27/34 2024-04-01T04:46:32.687 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x8 28/34 2024-04-01T04:46:34.989 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 29/34 2024-04-01T04:46:35.033 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 29/34 2024-04-01T04:46:35.627 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 29/34 2024-04-01T04:46:35.670 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 30/34 2024-04-01T04:46:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 30/34 2024-04-01T04:46:35.757 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T04:46:35.757 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T04:46:35.758 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:35.787 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 31/34 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 31/34 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-04-01T04:46:45.739 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-04-01T04:46:45.740 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-04-01T04:46:45.740 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:46.659 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 32/34 2024-04-01T04:46:46.683 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 32/34 2024-04-01T04:46:46.683 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:46:46.683 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T04:46:46.683 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T04:46:46.683 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T04:46:46.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:46.685 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T04:46:46.770 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-01T04:46:46.771 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T04:46:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T04:46:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 1/34 2024-04-01T04:46:47.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 3/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 4/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 5/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_6 6/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 7/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9. 8/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x8 9/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 10/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 11/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 12/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 13/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-01T04:46:47.893 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-01T04:46:47.894 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-01T04:46:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-01T04:46:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 31/34 2024-04-01T04:46:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-01T04:46:47.895 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 33/34 2024-04-01T04:46:48.892 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T04:46:48.893 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.894 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T04:46:48.895 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T04:46:48.896 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T04:46:48.896 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T04:46:48.896 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T04:46:48.896 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:48.896 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:46:49.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2024-04-01T04:46:49.790 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:29 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:46:49.953 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T04:46:49.954 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:46:49.955 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 48 M 2024-04-01T04:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 205 M 2024-04-01T04:46:49.956 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:46:50.242 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-01T04:46:50.334 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): libcephsqlite-19.0.0-1977.g384f31f3.el9. 636 kB/s | 163 kB 00:00 2024-04-01T04:46:50.359 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 300 kB 00:00 2024-04-01T04:46:50.385 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-01T04:46:50.601 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-04-01T04:46:54.027 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): ceph-test-19.0.0-1977.g384f31f3.el9.x86_ 12 MB/s | 47 MB 00:03 2024-04-01T04:46:54.030 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:46:54.030 INFO:teuthology.orchestra.run.smithi067.stdout:Total 12 MB/s | 48 MB 00:04 2024-04-01T04:46:54.045 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:46:54.068 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:46:54.068 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:46:54.465 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:46:54.466 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:46:55.071 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:46:55.137 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-01T04:46:55.177 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-01T04:46:55.209 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-01T04:46:55.252 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-01T04:46:55.280 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 5/6 2024-04-01T04:47:00.020 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 5/6 2024-04-01T04:47:00.099 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 6/6 2024-04-01T04:47:02.559 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 6/6 2024-04-01T04:47:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 1/6 2024-04-01T04:47:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2/6 2024-04-01T04:47:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-01T04:47:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-01T04:47:02.560 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-01T04:47:02.919 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:02.920 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:03.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2024-04-01T04:47:03.784 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:43 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:03.946 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:03.949 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-04-01T04:47:03.949 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:03.949 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-04-01T04:47:03.949 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:03.949 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-04-01T04:47:03.949 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T04:47:03.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T04:47:03.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2024-04-01T04:47:03.952 INFO:teuthology.orchestra.run.smithi067.stdout:Install 36 Packages 2024-04-01T04:47:03.953 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:03.953 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 30 M 2024-04-01T04:47:03.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 107 M 2024-04-01T04:47:03.954 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:04.748 INFO:teuthology.orchestra.run.smithi067.stdout:(1/36): ceph-19.0.0-1977.g384f31f3.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-04-01T04:47:05.099 INFO:teuthology.orchestra.run.smithi067.stdout:(2/36): ceph-mon-19.0.0-1977.g384f31f3.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-04-01T04:47:05.882 INFO:teuthology.orchestra.run.smithi067.stdout:(3/36): ceph-osd-19.0.0-1977.g384f31f3.el9.x86_ 21 MB/s | 17 MB 00:00 2024-04-01T04:47:05.932 INFO:teuthology.orchestra.run.smithi067.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1977.g384f 4.9 MB/s | 248 kB 00:00 2024-04-01T04:47:06.299 INFO:teuthology.orchestra.run.smithi067.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 509 kB/s | 187 kB 00:00 2024-04-01T04:47:06.466 INFO:teuthology.orchestra.run.smithi067.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 866 kB/s | 144 kB 00:00 2024-04-01T04:47:06.533 INFO:teuthology.orchestra.run.smithi067.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 474 kB/s | 32 kB 00:00 2024-04-01T04:47:06.617 INFO:teuthology.orchestra.run.smithi067.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 568 kB/s | 47 kB 00:00 2024-04-01T04:47:06.692 INFO:teuthology.orchestra.run.smithi067.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 516 kB/s | 39 kB 00:00 2024-04-01T04:47:06.815 INFO:teuthology.orchestra.run.smithi067.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 241 kB 00:00 2024-04-01T04:47:07.034 INFO:teuthology.orchestra.run.smithi067.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 5.2 MB/s | 1.1 MB 00:00 2024-04-01T04:47:07.093 INFO:teuthology.orchestra.run.smithi067.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-04-01T04:47:07.160 INFO:teuthology.orchestra.run.smithi067.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-04-01T04:47:07.227 INFO:teuthology.orchestra.run.smithi067.stdout:(14/36): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 909 kB/s | 2.4 MB 00:02 2024-04-01T04:47:07.252 INFO:teuthology.orchestra.run.smithi067.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.0 MB/s | 187 kB 00:00 2024-04-01T04:47:07.369 INFO:teuthology.orchestra.run.smithi067.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 882 kB/s | 125 kB 00:00 2024-04-01T04:47:07.861 INFO:teuthology.orchestra.run.smithi067.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 307 kB/s | 151 kB 00:00 2024-04-01T04:47:07.953 INFO:teuthology.orchestra.run.smithi067.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 475 kB/s | 43 kB 00:00 2024-04-01T04:47:07.970 INFO:teuthology.orchestra.run.smithi067.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-04-01T04:47:08.020 INFO:teuthology.orchestra.run.smithi067.stdout:(20/36): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 443 kB/s | 1.5 MB 00:03 2024-04-01T04:47:08.087 INFO:teuthology.orchestra.run.smithi067.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-01T04:47:08.187 INFO:teuthology.orchestra.run.smithi067.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-04-01T04:47:08.212 INFO:teuthology.orchestra.run.smithi067.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 56 kB/s | 11 kB 00:00 2024-04-01T04:47:08.238 INFO:teuthology.orchestra.run.smithi067.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 117 kB/s | 18 kB 00:00 2024-04-01T04:47:08.271 INFO:teuthology.orchestra.run.smithi067.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-04-01T04:47:08.321 INFO:teuthology.orchestra.run.smithi067.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-04-01T04:47:08.347 INFO:teuthology.orchestra.run.smithi067.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-04-01T04:47:08.372 INFO:teuthology.orchestra.run.smithi067.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-04-01T04:47:08.414 INFO:teuthology.orchestra.run.smithi067.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 856 kB/s | 79 kB 00:00 2024-04-01T04:47:08.448 INFO:teuthology.orchestra.run.smithi067.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-01T04:47:08.481 INFO:teuthology.orchestra.run.smithi067.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2024-04-01T04:47:08.506 INFO:teuthology.orchestra.run.smithi067.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 978 kB/s | 90 kB 00:00 2024-04-01T04:47:08.531 INFO:teuthology.orchestra.run.smithi067.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-04-01T04:47:08.573 INFO:teuthology.orchestra.run.smithi067.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-01T04:47:08.624 INFO:teuthology.orchestra.run.smithi067.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-04-01T04:47:08.650 INFO:teuthology.orchestra.run.smithi067.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 170 kB/s | 20 kB 00:00 2024-04-01T04:47:08.653 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:08.653 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.4 MB/s | 30 MB 00:04 2024-04-01T04:47:08.679 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:08.725 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:08.726 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:09.160 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:09.160 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:10.523 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:10.585 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-01T04:47:10.627 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-01T04:47:10.654 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-01T04:47:10.730 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-01T04:47:10.778 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-01T04:47:10.811 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-01T04:47:10.907 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-01T04:47:10.984 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-01T04:47:11.031 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-01T04:47:11.106 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-01T04:47:11.166 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-01T04:47:11.203 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-01T04:47:11.262 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-01T04:47:11.325 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-01T04:47:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-01T04:47:11.573 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-01T04:47:11.612 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-01T04:47:11.646 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-01T04:47:11.676 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-01T04:47:11.757 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-01T04:47:11.795 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T04:47:11.829 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-01T04:47:11.869 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-01T04:47:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-01T04:47:11.946 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-01T04:47:12.042 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-01T04:47:12.086 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-01T04:47:12.162 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-01T04:47:12.312 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-01T04:47:12.443 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T04:47:12.611 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9. 31/36 2024-04-01T04:47:12.640 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 32/36 2024-04-01T04:47:12.665 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 32/36 2024-04-01T04:47:12.666 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:47:12.666 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T04:47:12.666 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T04:47:12.666 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T04:47:12.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:14.200 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 33/36 2024-04-01T04:47:14.225 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 33/36 2024-04-01T04:47:14.225 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:47:14.225 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T04:47:14.225 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T04:47:14.225 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T04:47:14.226 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:14.692 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 34/36 2024-04-01T04:47:14.715 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 34/36 2024-04-01T04:47:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:47:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T04:47:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T04:47:14.715 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T04:47:14.716 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:14.951 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 35/36 2024-04-01T04:47:14.973 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 35/36 2024-04-01T04:47:14.974 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:47:14.974 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T04:47:14.974 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T04:47:14.974 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T04:47:14.974 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:15.042 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 36/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 36/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 3/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 4/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 5/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9. 6/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-01T04:47:16.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-01T04:47:16.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-01T04:47:16.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-01T04:47:17.670 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-01T04:47:17.670 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:17.670 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:17.670 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:17.670 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T04:47:17.671 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T04:47:17.672 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:17.673 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:18.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2024-04-01T04:47:18.598 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:58 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:18.699 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:47:18.753 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:18.754 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:47:18.754 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:18.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2024-04-01T04:47:19.392 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:59 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:19.548 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:19.548 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 748 k 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 755 k 2024-04-01T04:47:19.549 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:19.822 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-19.0.0-1977.g384f31f3.el9.noarch.rpm 2.7 MB/s | 748 kB 00:00 2024-04-01T04:47:19.823 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:19.823 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.7 MB/s | 748 kB 00:00 2024-04-01T04:47:19.823 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:19.828 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:19.828 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:19.835 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:19.835 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:19.914 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:20.372 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 1/1 2024-04-01T04:47:20.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 1/1 2024-04-01T04:47:20.856 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 1/1 2024-04-01T04:47:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 1/1 2024-04-01T04:47:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:21.124 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:21.124 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:21.124 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:21.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2024-04-01T04:47:21.940 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:01 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:22.097 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:22.097 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 145 k 2024-04-01T04:47:22.098 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 435 k 2024-04-01T04:47:22.099 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:22.478 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-19.0.0-1977.g384f31 382 kB/s | 145 kB 00:00 2024-04-01T04:47:22.478 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:22.478 INFO:teuthology.orchestra.run.smithi067.stdout:Total 380 kB/s | 145 kB 00:00 2024-04-01T04:47:22.479 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:22.487 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:22.487 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:22.546 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:22.547 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:22.702 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:22.753 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3. 1/1 2024-04-01T04:47:22.778 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3. 1/1 2024-04-01T04:47:22.778 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:47:22.778 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T04:47:22.778 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3. 1/1 2024-04-01T04:47:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:23.758 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:23.758 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:47:23.759 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:23.759 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:23.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2024-04-01T04:47:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:04 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:24.667 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:47:24.722 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:24.723 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:47:24.723 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:24.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2024-04-01T04:47:25.356 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:05 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:25.514 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:25.515 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T04:47:25.516 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-04-01T04:47:25.517 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:25.518 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 8.6 M 2024-04-01T04:47:25.518 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 103 M 2024-04-01T04:47:25.518 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:26.114 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1977.g38 160 kB/s | 31 kB 00:00 2024-04-01T04:47:26.281 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1977.g384 42 kB/s | 15 kB 00:00 2024-04-01T04:47:26.465 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-04-01T04:47:26.490 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-04-01T04:47:26.657 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-04-01T04:47:26.734 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1977.g384f31f 4.3 MB/s | 3.5 MB 00:00 2024-04-01T04:47:26.900 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-04-01T04:47:26.941 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-04-01T04:47:27.042 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-04-01T04:47:27.084 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 787 kB/s | 144 kB 00:00 2024-04-01T04:47:27.167 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-04-01T04:47:27.351 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-04-01T04:47:27.353 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:27.353 INFO:teuthology.orchestra.run.smithi067.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-04-01T04:47:27.372 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:27.391 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:27.391 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:27.596 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:28.136 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:28.210 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-01T04:47:28.245 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-01T04:47:28.292 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-01T04:47:28.322 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-01T04:47:28.424 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-01T04:47:28.554 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-01T04:47:28.645 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-01T04:47:28.694 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-01T04:47:28.721 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-01T04:47:28.753 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9 10/12 2024-04-01T04:47:29.848 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el 11/12 2024-04-01T04:47:29.878 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noa 12/12 2024-04-01T04:47:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noa 12/12 2024-04-01T04:47:31.256 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el 1/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noa 2/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9 3/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-01T04:47:31.257 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T04:47:31.600 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T04:47:31.601 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T04:47:31.601 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T04:47:31.601 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T04:47:31.601 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:31.601 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:31.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-01T04:47:32.466 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:12 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-04-01T04:47:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:32.628 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:32.629 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2024-04-01T04:47:32.629 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2024-04-01T04:47:32.629 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:32.629 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 37 M 2024-04-01T04:47:32.629 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 209 M 2024-04-01T04:47:32.629 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:33.167 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 173 kB/s | 32 kB 00:00 2024-04-01T04:47:33.217 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 344 kB/s | 17 kB 00:00 2024-04-01T04:47:33.276 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 587 kB/s | 34 kB 00:00 2024-04-01T04:47:33.409 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.0 MB/s | 3.0 MB 00:00 2024-04-01T04:47:33.760 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.8 MB/s | 4.7 MB 00:00 2024-04-01T04:47:33.826 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-04-01T04:47:33.851 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-04-01T04:47:34.285 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-04-01T04:47:34.502 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 903 kB/s | 195 kB 00:00 2024-04-01T04:47:35.152 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 377 kB/s | 245 kB 00:00 2024-04-01T04:47:35.403 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 3.1 MB/s | 7.4 MB 00:02 2024-04-01T04:47:35.445 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 504 kB/s | 803 kB 00:01 2024-04-01T04:47:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:35.447 INFO:teuthology.orchestra.run.smithi067.stdout:Total 13 MB/s | 37 MB 00:02 2024-04-01T04:47:35.492 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:35.518 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:35.518 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:35.901 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:35.902 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:36.630 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:36.718 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-01T04:47:36.749 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-01T04:47:36.854 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-01T04:47:36.885 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-01T04:47:37.070 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T04:47:37.101 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T04:47:37.189 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-01T04:47:37.865 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-01T04:47:37.984 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-01T04:47:39.287 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-01T04:47:40.361 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-01T04:47:40.391 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f3 12/12 2024-04-01T04:47:42.935 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f3 12/12 2024-04-01T04:47:42.935 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f3 1/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-01T04:47:42.936 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-01T04:47:43.366 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-01T04:47:43.366 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:43.366 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:43.367 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:43.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2024-04-01T04:47:44.328 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:24 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:44.486 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T04:47:44.488 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2024-04-01T04:47:44.489 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:44.490 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.0 M 2024-04-01T04:47:44.490 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2024-04-01T04:47:44.490 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:44.701 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-04-01T04:47:44.725 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 99 kB/s | 18 kB 00:00 2024-04-01T04:47:44.750 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): ceph-mgr-rook-19.0.0-1977.g384f31f3.el9 238 kB/s | 49 kB 00:00 2024-04-01T04:47:44.892 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 909 kB/s | 174 kB 00:00 2024-04-01T04:47:44.917 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 688 kB/s | 132 kB 00:00 2024-04-01T04:47:44.959 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 211 kB 00:00 2024-04-01T04:47:44.984 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 465 kB/s | 43 kB 00:00 2024-04-01T04:47:45.201 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-04-01T04:47:45.243 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-04-01T04:47:45.360 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 540 kB/s | 203 kB 00:00 2024-04-01T04:47:45.385 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-04-01T04:47:45.427 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-04-01T04:47:45.786 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-04-01T04:47:45.790 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-04-01T04:47:45.791 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:45.806 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:45.807 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:45.907 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:45.909 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:46.425 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:46.606 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-01T04:47:46.664 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-01T04:47:46.719 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-01T04:47:46.754 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-01T04:47:46.797 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-01T04:47:46.894 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-01T04:47:47.004 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-01T04:47:47.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-01T04:47:47.819 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-01T04:47:47.869 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-01T04:47:47.903 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-01T04:47:47.951 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-01T04:47:47.981 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 13/13 2024-04-01T04:47:48.826 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 13/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 1/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-01T04:47:48.827 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-01T04:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-01T04:47:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T04:47:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T04:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T04:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:49.183 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:49.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2024-04-01T04:47:50.018 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:30 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:50.178 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:50.179 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2024-04-01T04:47:50.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:50.181 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 6.8 M 2024-04-01T04:47:50.181 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2024-04-01T04:47:50.181 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:50.878 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1977.g384f31f3.e 697 kB/s | 139 kB 00:00 2024-04-01T04:47:51.370 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.4 MB/s | 5.8 MB 00:00 2024-04-01T04:47:51.562 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 281 kB/s | 248 kB 00:00 2024-04-01T04:47:51.895 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 539 kB/s | 548 kB 00:01 2024-04-01T04:47:52.054 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 84 kB/s | 58 kB 00:00 2024-04-01T04:47:52.237 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 75 kB/s | 51 kB 00:00 2024-04-01T04:47:52.241 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:52.241 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.3 MB/s | 6.8 MB 00:02 2024-04-01T04:47:52.255 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:52.266 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:52.266 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:52.359 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:52.360 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:52.662 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:52.833 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-01T04:47:52.890 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-01T04:47:53.451 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-01T04:47:53.544 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-01T04:47:53.606 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-01T04:47:53.635 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 6/6 2024-04-01T04:47:54.457 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 6/6 2024-04-01T04:47:54.457 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 1/6 2024-04-01T04:47:54.457 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-01T04:47:54.457 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-01T04:47:54.457 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-01T04:47:54.457 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T04:47:54.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:54.766 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:55.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2024-04-01T04:47:55.578 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:35 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:55.735 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-04-01T04:47:55.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 968 k 2024-04-01T04:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.8 M 2024-04-01T04:47:55.737 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:55.947 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 374 kB/s | 79 kB 00:00 2024-04-01T04:47:56.372 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-19.0.0-1977.g384f31f3.el9.x86_ 1.4 MB/s | 889 kB 00:00 2024-04-01T04:47:56.373 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:56.374 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.5 MB/s | 968 kB 00:00 2024-04-01T04:47:56.374 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:56.395 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:56.396 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:56.472 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:56.473 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:56.659 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:56.803 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-01T04:47:56.876 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2/2 2024-04-01T04:47:57.609 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2/2 2024-04-01T04:47:57.609 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 1/2 2024-04-01T04:47:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-01T04:47:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:57.905 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:47:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-01T04:47:57.905 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:57.905 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:47:58.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2024-04-01T04:47:58.778 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:38 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:47:58.937 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:47:58.938 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:47:58.939 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2024-04-01T04:47:58.939 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:47:58.939 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 332 k 2024-04-01T04:47:58.939 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.5 M 2024-04-01T04:47:58.939 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:47:59.176 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 290 kB/s | 69 kB 00:00 2024-04-01T04:47:59.201 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-volume-19.0.0-1977.g384f31f3.el9.no 1.0 MB/s | 264 kB 00:00 2024-04-01T04:47:59.203 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:47:59.203 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-04-01T04:47:59.203 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:47:59.212 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:47:59.212 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:47:59.252 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:47:59.253 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:47:59.402 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:47:59.590 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-01T04:47:59.620 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2/2 2024-04-01T04:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2/2 2024-04-01T04:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T04:47:59.636 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:00.378 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 1/2 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:00.712 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:01.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2024-04-01T04:48:01.628 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:41 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:01.787 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:01.787 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:01.787 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 127 k 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 456 k 2024-04-01T04:48:01.788 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:48:01.993 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-19.0.0-1977.g384f31f3.el9.x86_64 620 kB/s | 127 kB 00:00 2024-04-01T04:48:01.993 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:48:01.993 INFO:teuthology.orchestra.run.smithi067.stdout:Total 616 kB/s | 127 kB 00:00 2024-04-01T04:48:01.994 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:48:02.000 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:48:02.000 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:48:02.058 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:48:02.058 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:48:02.215 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:48:02.310 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:02.745 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:03.027 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:03.027 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:03.027 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:48:03.028 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:48:03.028 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:03.028 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:03.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2024-04-01T04:48:03.857 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:43 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:03.961 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:04.019 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:04.020 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:04.020 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:04.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2024-04-01T04:48:04.659 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:44 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:04.819 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:04.819 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:04.819 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 32 k 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 142 k 2024-04-01T04:48:04.820 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:48:05.196 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-19.0.0-1977.g384f31f3.el9.x86_6 84 kB/s | 32 kB 00:00 2024-04-01T04:48:05.197 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:48:05.197 INFO:teuthology.orchestra.run.smithi067.stdout:Total 84 kB/s | 32 kB 00:00 2024-04-01T04:48:05.197 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:48:05.200 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:48:05.201 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:48:05.235 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:48:05.235 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:48:05.352 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:48:05.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:05.824 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:06.086 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:06.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:06.086 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:48:06.086 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:48:06.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:06.086 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:06.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2024-04-01T04:48:06.798 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:46 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:06.898 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:06.952 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:06.954 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:06.954 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:06.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2024-04-01T04:48:07.573 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:47 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:07.673 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:07.728 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:07.775 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2024-04-01T04:48:08.355 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:48 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:08.457 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:08.512 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:08.513 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:08.513 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:08.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2024-04-01T04:48:09.138 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:49 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:09.238 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:09.293 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:09.294 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:09.294 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:09.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2024-04-01T04:48:09.912 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:49 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:10.012 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:10.067 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:10.068 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:10.068 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:10.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2024-04-01T04:48:10.693 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:50 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:10.795 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 is already installed. 2024-04-01T04:48:10.850 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:10.851 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-04-01T04:48:10.851 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:11.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2024-04-01T04:48:11.577 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:51 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:11.733 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:11.734 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:48:11.735 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:11.735 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 85 k 2024-04-01T04:48:11.735 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 231 k 2024-04-01T04:48:11.735 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:48:12.057 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-19.0.0-1977.g384f31f3.el9.x86_64.rpm 264 kB/s | 85 kB 00:00 2024-04-01T04:48:12.058 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:48:12.058 INFO:teuthology.orchestra.run.smithi067.stdout:Total 263 kB/s | 85 kB 00:00 2024-04-01T04:48:12.058 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:48:12.066 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:48:12.066 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:48:12.123 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:48:12.124 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:48:12.272 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:48:12.385 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:12.885 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:13.168 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:48:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:48:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:13.169 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:13.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2024-04-01T04:48:13.982 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:53 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:14.140 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:14.141 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:48:14.142 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:14.142 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:48:14.142 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:14.142 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.1 M 2024-04-01T04:48:14.142 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 13 M 2024-04-01T04:48:14.142 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:48:15.052 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-19.0.0-1977.g384f31f3.el9.x86_64.rpm 3.4 MB/s | 3.1 MB 00:00 2024-04-01T04:48:15.052 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:48:15.053 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.4 MB/s | 3.1 MB 00:00 2024-04-01T04:48:15.053 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:48:15.064 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:48:15.064 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:48:15.144 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:48:15.145 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:48:15.595 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:48:15.661 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:15.685 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:15.685 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T04:48:15.685 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T04:48:15.685 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T04:48:15.686 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T04:48:15.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:16.834 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:16.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:16.834 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:48:16.834 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:48:16.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:16.834 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:17.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2024-04-01T04:48:17.677 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:57 ago on Mon 01 Apr 2024 04:46:20 AM UTC. 2024-04-01T04:48:17.833 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2024-04-01T04:48:17.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:17.835 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 171 k 2024-04-01T04:48:17.835 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 494 k 2024-04-01T04:48:17.835 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-04-01T04:48:18.222 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-19.0.0-1977.g384f31f3.el9.x86_64.rpm 443 kB/s | 171 kB 00:00 2024-04-01T04:48:18.222 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-04-01T04:48:18.223 INFO:teuthology.orchestra.run.smithi067.stdout:Total 441 kB/s | 171 kB 00:00 2024-04-01T04:48:18.223 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-04-01T04:48:18.231 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-04-01T04:48:18.231 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-04-01T04:48:18.289 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-04-01T04:48:18.290 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-04-01T04:48:18.439 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-04-01T04:48:18.548 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:18.973 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:19.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 1/1 2024-04-01T04:48:19.261 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-04-01T04:48:19.261 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T04:48:19.261 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-01T04:48:19.261 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-04-01T04:48:19.529 DEBUG:teuthology.parallel:result is None 2024-04-01T04:48:19.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T04:48:19.677 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T04:48:19.697 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-1977.g384f31f3.el9 2024-04-01T04:48:19.698 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-04-01T04:48:19.698 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-04-01T04:48:19.700 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-01T04:48:19.700 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:48:19.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T04:48:19.765 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-01T04:48:19.765 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:48:19.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T04:48:19.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T04:48:19.897 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-01T04:48:19.898 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:48:19.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T04:48:19.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T04:48:20.031 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-01T04:48:20.031 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:48:20.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T04:48:20.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T04:48:20.164 INFO:teuthology.run_tasks:Running task workunit... 2024-04-01T04:48:20.177 INFO:tasks.workunit:Pulling workunits from ref 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T04:48:20.177 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-01T04:48:20.177 INFO:tasks.workunit:timeout=3h 2024-04-01T04:48:20.177 INFO:tasks.workunit:cleanup=True 2024-04-01T04:48:20.177 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-01T04:48:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T04:48:20.217 INFO:teuthology.orchestra.run.smithi067.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-01T04:48:20.218 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-01T04:48:20.272 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-01T04:48:20.272 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-01T04:48:20.327 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T04:48:20.382 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-01T04:49:26.858 INFO:tasks.workunit.client.0.smithi067.stderr:Updating files: 99% (12375/12435) Updating files: 100% (12435/12435) Updating files: 100% (12435/12435), done. 2024-04-01T04:49:27.601 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to '384f31f3d3aee7f8ba3f60846b8844c049165583'. 2024-04-01T04:49:27.601 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2024-04-01T04:49:27.602 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.603 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-01T04:49:27.603 INFO:tasks.workunit.client.0.smithi067.stderr: 2024-04-01T04:49:27.603 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at 384f31f3d3a Merge pull request #56593 from zdover23/wip-doc-2024-03-31-backport-56592-to-squid 2024-04-01T04:49:27.606 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-01T04:49:27.661 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-01T04:49:27.662 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-01T04:49:27.716 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-04-01T04:49:27.717 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-04-01T04:49:27.717 DEBUG:teuthology.orchestra.run.smithi067:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-04-01T04:49:27.783 INFO:tasks.workunit.client.0.smithi067.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-01T04:49:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-04-01T04:49:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-04-01T04:49:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-04-01T04:49:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-04-01T04:49:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-04-01T04:49:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-04-01T04:49:27.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-04-01T04:49:27.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none ' 2024-04-01T04:49:27.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-04-01T04:49:27.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-04-01T04:49:27.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-04-01T04:49:27.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:49:27.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:49:27.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:49:27.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:49:27.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:49:27.797 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:49:27.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:49:27.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:49:27.798 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:49:27.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:49:27.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:49:27.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:49:27.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:49:27.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:49:27.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:49:27.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-04-01T04:49:27.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:49:27.802 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:49:27.802 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:49:27.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28982 2024-04-01T04:49:27.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:49:27.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:49:27.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-04-01T04:49:27.804 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T04:49:27.804 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:49:27.804 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:49:27.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.28982 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-04-01T04:49:27.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-04-01T04:49:27.807 INFO:tasks.workunit.client.0.smithi067.stdout:Dir: td/osd-mapper 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-04-01T04:49:27.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-04-01T04:49:27.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-04-01T04:49:27.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-04-01T04:49:27.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-04-01T04:49:27.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-04-01T04:49:29.474 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 36d77667-401b-4b37-812b-93c231d69f78 2024-04-01T04:49:29.726 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:49:29.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.813+0000 7f821c01e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:49:29.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.815+0000 7f821c01e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:49:29.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.815+0000 7f821c01e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:49:29.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.816+0000 7f821c01e640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-04-01T04:49:29.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.816+0000 7f821c01e640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-04-01T04:49:29.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.816+0000 7f821c01e640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-04-01T04:49:29.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:29.885+0000 7f821c01e640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-04-01T04:49:35.733 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T04:49:36.097 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T04:49:36.126 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:36.125+0000 7fba62d47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:49:36.127 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:36.127+0000 7fba62d47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:49:36.128 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:36.128+0000 7fba62d47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:49:36.377 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:49:36.642 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:36.642+0000 7fba62d47640 -1 Falling back to public interface 2024-04-01T04:49:37.660 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:49:38.949 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:49:38.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:49:38.989+0000 7fba62d47640 -1 osd.0 0 log_to_monitors true 2024-04-01T04:49:40.424 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:49:41.712 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T04:49:43.004 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T04:49:44.299 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T04:49:45.592 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T04:49:46.921 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T04:49:48.213 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T04:49:49.560 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T04:49:50.884 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T04:49:52.174 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T04:49:53.587 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T04:49:55.206 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T04:49:56.832 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T04:49:58.619 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T04:50:00.824 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T04:50:02.603 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T04:50:02.893 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3263531302,v1:127.0.0.1:6803/3263531302] [v2:127.0.0.1:6804/3263531302,v1:127.0.0.1:6805/3263531302] exists,up 36d77667-401b-4b37-812b-93c231d69f78 2024-04-01T04:50:02.897 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 7bbba687-a568-4cd7-8426-901553c6d11b 2024-04-01T04:50:03.322 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:50:03.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.363+0000 7f32ff622640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:03.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.406+0000 7f32ff622640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:03.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.407+0000 7f32ff622640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:03.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.408+0000 7f32ff622640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-04-01T04:50:03.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.408+0000 7f32ff622640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-04-01T04:50:03.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.408+0000 7f32ff622640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-04-01T04:50:03.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:03.481+0000 7f32ff622640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-04-01T04:50:09.238 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T04:50:09.624 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T04:50:09.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:09.652+0000 7fc8e03bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:09.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:09.653+0000 7fc8e03bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:09.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:09.655+0000 7fc8e03bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:09.904 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:50:10.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:10.172+0000 7fc8e03bd640 -1 Falling back to public interface 2024-04-01T04:50:11.184 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:50:12.473 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:50:12.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:12.513+0000 7fc8e03bd640 -1 osd.1 0 log_to_monitors true 2024-04-01T04:50:13.822 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:50:15.115 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T04:50:16.601 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T04:50:17.891 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T04:50:19.184 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T04:50:20.594 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T04:50:21.885 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T04:50:23.291 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T04:50:24.584 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T04:50:26.207 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T04:50:27.685 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T04:50:29.128 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T04:50:30.513 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T04:50:32.225 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T04:50:33.674 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T04:50:35.381 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T04:50:35.712 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1920965909,v1:127.0.0.1:6811/1920965909] [v2:127.0.0.1:6812/1920965909,v1:127.0.0.1:6813/1920965909] exists,up 7bbba687-a568-4cd7-8426-901553c6d11b 2024-04-01T04:50:35.744 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 9c4945d3-7284-465a-ad54-3b37ad9750eb 2024-04-01T04:50:36.067 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:50:36.112 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.110+0000 7fa6b13bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:36.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.142+0000 7fa6b13bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:36.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.143+0000 7fa6b13bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:36.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.143+0000 7fa6b13bd640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-04-01T04:50:36.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.143+0000 7fa6b13bd640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-04-01T04:50:36.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.143+0000 7fa6b13bd640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-04-01T04:50:36.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:36.217+0000 7fa6b13bd640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-04-01T04:50:42.301 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T04:50:42.664 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T04:50:42.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:42.692+0000 7f83b1516640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:42.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:42.694+0000 7f83b1516640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:42.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:42.695+0000 7f83b1516640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:50:42.948 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:50:43.236 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:43.235+0000 7f83b1516640 -1 Falling back to public interface 2024-04-01T04:50:44.227 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:50:45.519 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:50:45.589 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:50:45.588+0000 7f83b1516640 -1 osd.2 0 log_to_monitors true 2024-04-01T04:50:46.861 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:50:48.154 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T04:50:49.479 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T04:50:50.793 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T04:50:52.133 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T04:50:53.493 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T04:50:54.804 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T04:50:56.133 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T04:50:57.483 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T04:50:58.987 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T04:51:00.696 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T04:51:02.049 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T04:51:03.533 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T04:51:05.228 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T04:51:06.898 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T04:51:08.356 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T04:51:08.646 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/735125775,v1:127.0.0.1:6819/735125775] [v2:127.0.0.1:6820/735125775,v1:127.0.0.1:6821/735125775] exists,up 9c4945d3-7284-465a-ad54-3b37ad9750eb 2024-04-01T04:51:09.137 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'test' created 2024-04-01T04:51:11.431 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836501 2024-04-01T04:51:14.302 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672972 2024-04-01T04:51:14.584 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509444 2024-04-01T04:51:17.196 INFO:tasks.workunit.client.0.smithi067.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-04-01T04:51:17.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-04-01T04:51:17.313 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0: { 2024-04-01T04:51:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-04-01T04:51:17.314 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:17.328 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1: { 2024-04-01T04:51:17.328 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-04-01T04:51:17.328 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:17.342 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2: { 2024-04-01T04:51:17.342 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-04-01T04:51:17.342 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:17.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-04-01T04:51:17.472 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0: { 2024-04-01T04:51:17.472 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-04-01T04:51:17.472 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:17.486 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1: { 2024-04-01T04:51:17.486 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-04-01T04:51:17.486 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:17.500 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2: { 2024-04-01T04:51:17.500 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-04-01T04:51:17.500 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:17.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-04-01T04:51:17.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-04-01T04:51:17.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-04-01T04:51:17.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-04-01T04:51:17.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-04-01T04:51:17.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-04-01T04:51:17.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-04-01T04:51:17.806 INFO:tasks.workunit.client.0.smithi067.stdout:created pool test snap snap01 2024-04-01T04:51:18.049 INFO:tasks.workunit.client.0.smithi067.stdout:created pool test snap snap02 2024-04-01T04:51:18.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-04-01T04:51:18.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-04-01T04:51:18.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-04-01T04:51:18.510 INFO:tasks.workunit.client.0.smithi067.stdout:created pool test snap snap13 2024-04-01T04:51:18.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-04-01T04:51:18.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-04-01T04:51:18.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-04-01T04:51:18.971 INFO:tasks.workunit.client.0.smithi067.stdout:created pool test snap snap24 2024-04-01T04:51:19.239 INFO:tasks.workunit.client.0.smithi067.stdout:created pool test snap snap25 2024-04-01T04:51:19.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 13687 2024-04-01T04:51:19.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-04-01T04:51:19.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-04-01T04:51:19.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-04-01T04:51:19.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-04-01T04:51:19.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-04-01T04:51:19.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-04-01T04:51:20.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-04-01T04:51:20.120 INFO:tasks.workunit.client.0.smithi067.stdout:pgid is 1.3 (primary: osd.1) 2024-04-01T04:51:20.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-04-01T04:51:20.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-04-01T04:51:20.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-04-01T04:51:20.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-04-01T04:51:20.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-04-01T04:51:20.385 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs_brief 2024-04-01T04:51:20.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-04-01T04:51:20.397 INFO:tasks.workunit.client.0.smithi067.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:51:20.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.28982/ceph-osd.1.asok 2024-04-01T04:51:20.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-04-01T04:51:20.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.28982/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-04-01T04:51:20.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:20.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-04-01T04:51:20.517 INFO:tasks.workunit.client.0.smithi067.stdout:{"success":true}{ 2024-04-01T04:51:20.517 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "objxxx", 2024-04-01T04:51:20.517 INFO:tasks.workunit.client.0.smithi067.stdout: "clones": [ 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [ 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 1, 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap01" 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap02" 2024-04-01T04:51:20.518 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 6, 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "overlaps": [] 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [ 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap13" 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 5, 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: "overlaps": [] 2024-04-01T04:51:20.519 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [ 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 4, 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap24" 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:51:20.520 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap25" 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 6, 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "overlaps": [] 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "id": "head", 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [], 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 6 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-04-01T04:51:20.521 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:20.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-04-01T04:51:20.639 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:51:20.639 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": true, 2024-04-01T04:51:20.639 INFO:tasks.workunit.client.0.smithi067.stdout: "must": true, 2024-04-01T04:51:20.639 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "0.000000" 2024-04-01T04:51:20.639 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:51:20.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-04-01T04:51:23.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-04-01T04:51:10.379927+0000 28'7 28:37 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:51:10.966421+0000 0'0 25:24 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:51:10.039854+0000 0'0 25:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:51:10.379858+0000 0'0 28:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:51:23.923 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:51:23.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-04-01T04:51:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-04-01T04:51:24.199 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:51:24.199 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-04-01T04:51:10.379927+0000 28'7 28:37 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:24.200 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:51:10.966421+0000 0'0 25:24 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:24.200 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:51:10.039854+0000 0'0 25:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:24.200 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:51:10.379858+0000 0'0 28:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:51:24.200 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:24.200 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:51:24.200 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:51:24.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-04-01T04:51:24.662 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T04:51:24.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-04-01T04:51:25.165 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T04:51:25.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-04-01T04:51:30.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-04-01T04:51:30.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-04-01T04:51:30.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:51:30.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:51:30.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:51:30.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:51:30.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:51:30.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:51:30.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:30.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bdev(0x55b84f020000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 0 bdev(0x55b84f020000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bdev(0x55b84f020000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bdev(0x55b84f020700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 0 bdev(0x55b84f020700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bdev(0x55b84f020700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-04-01T04:51:30.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.512+0000 7fad09851980 1 bluefs mount 2024-04-01T04:51:30.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.517+0000 7fad09851980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:30.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.544+0000 7fad09851980 1 bluefs mount shared_bdev_used = 0 2024-04-01T04:51:30.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.544+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:30.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.546+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.546+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.546+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.546+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.546+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.546+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.547+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.547+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.547+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.547+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.547+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:30.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.547+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:30.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.579+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:30.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.579+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-04-01T04:51:30.580 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.579+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.579+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.579+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.580+0000 7fad09851980 1 freelist init 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.580+0000 7fad09851980 1 freelist _read_cfg 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.580+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 8.77383e-07 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.581+0000 7fad09851980 1 bluefs umount 2024-04-01T04:51:30.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.581+0000 7fad09851980 1 bdev(0x55b84f020700 td/osd-mapper/0/block) close 2024-04-01T04:51:30.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.782+0000 7fad09851980 1 bdev(0x55b84f020700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-04-01T04:51:30.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.782+0000 7fad09851980 0 bdev(0x55b84f020700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-04-01T04:51:30.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.782+0000 7fad09851980 1 bdev(0x55b84f020700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:30.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.782+0000 7fad09851980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-04-01T04:51:30.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.782+0000 7fad09851980 1 bluefs mount 2024-04-01T04:51:30.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.782+0000 7fad09851980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:30.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.788+0000 7fad09851980 1 bluefs mount shared_bdev_used = 27459584 2024-04-01T04:51:30.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.788+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.790+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.790+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:30.791+0000 7fad09851980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:31.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:31.692+0000 7fad09851980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 0b 05 58 02 00 00 55 24 bd 08 99 fb 4d 64 82 f9 |..X...U$....Md..| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 36 77 87 50 08 f4 00 00 00 00 1e 00 00 00 00 00 |6w.P............| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:* 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-04-01T04:51:31.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-04-01T04:51:31.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-04-01T04:51:31.697 INFO:tasks.workunit.client.0.smithi067.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 36 d7 76 67 40 1b |re........6.vg@.| 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:00000230 4b 37 81 2b 93 c2 31 d6 9f 78 00 00 00 00 00 00 |K7.+..1..x......| 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:00000240 00 00 1e 00 00 00 fa 3c 0a 66 98 c9 38 10 01 00 |.......<.f..8...| 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:0000025e 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:31.698 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:0000003f 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:31.699 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:31.700 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.701 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.702 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-04-01T04:51:31.703 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.703 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-04-01T04:51:31.703 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:31.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:31.693+0000 7fad09851980 1 bluefs umount 2024-04-01T04:51:31.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:31.693+0000 7fad09851980 1 bdev(0x55b84f020700 td/osd-mapper/0/block) close 2024-04-01T04:51:31.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:31.913+0000 7fad09851980 1 freelist shutdown 2024-04-01T04:51:31.913 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:31.913+0000 7fad09851980 1 bdev(0x55b84f020000 td/osd-mapper/0/block) close 2024-04-01T04:51:32.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:32.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-04-01T04:51:32.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-04-01T04:51:32.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.056+0000 7f1495e2f980 1 bdev(0x55ce1d356000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.056+0000 7f1495e2f980 0 bdev(0x55ce1d356000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 1 bdev(0x55ce1d356000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 1 bdev(0x55ce1d356700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 0 bdev(0x55ce1d356700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 1 bdev(0x55ce1d356700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-04-01T04:51:32.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.057+0000 7f1495e2f980 1 bluefs mount 2024-04-01T04:51:32.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.069+0000 7f1495e2f980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:32.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.077+0000 7f1495e2f980 1 bluefs mount shared_bdev_used = 0 2024-04-01T04:51:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.077+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.079+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.079+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.079+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.079+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.079+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:32.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:32.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.080+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 freelist init 2024-04-01T04:51:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.112+0000 7f1495e2f980 1 freelist _read_cfg 2024-04-01T04:51:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.113+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.34059e-07 2024-04-01T04:51:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.113+0000 7f1495e2f980 1 bluefs umount 2024-04-01T04:51:32.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.113+0000 7f1495e2f980 1 bdev(0x55ce1d356700 td/osd-mapper/2/block) close 2024-04-01T04:51:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.322+0000 7f1495e2f980 1 bdev(0x55ce1d356700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-04-01T04:51:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.322+0000 7f1495e2f980 0 bdev(0x55ce1d356700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-04-01T04:51:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.322+0000 7f1495e2f980 1 bdev(0x55ce1d356700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.322+0000 7f1495e2f980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-04-01T04:51:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.322+0000 7f1495e2f980 1 bluefs mount 2024-04-01T04:51:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.322+0000 7f1495e2f980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.328+0000 7f1495e2f980 1 bluefs mount shared_bdev_used = 27459584 2024-04-01T04:51:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.328+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.330+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.331+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.331+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.331+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.331+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:32.331+0000 7f1495e2f980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:33.113 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:33.113+0000 7f1495e2f980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:33.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:33.114+0000 7f1495e2f980 1 bluefs umount 2024-04-01T04:51:33.114 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:33.114+0000 7f1495e2f980 1 bdev(0x55ce1d356700 td/osd-mapper/2/block) close 2024-04-01T04:51:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:33.325+0000 7f1495e2f980 1 freelist shutdown 2024-04-01T04:51:33.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:33.325+0000 7f1495e2f980 1 bdev(0x55ce1d356000 td/osd-mapper/2/block) close 2024-04-01T04:51:33.573 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:33.573 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:33.573 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:33.573 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:33.573 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:33.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:33.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-04-01T04:51:33.574 INFO:tasks.workunit.client.0.smithi067.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-04-01T04:51:33.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:33.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-04-01T04:51:34.267 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-04-01T04:51:34.267 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 0b 05 58 02 00 00 55 24 bd 08 99 fb 4d 64 82 f9 |..X...U$....Md..| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 36 77 87 50 08 f4 02 00 00 00 1e 00 00 00 00 00 |6w.P............| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:* 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-04-01T04:51:34.268 INFO:tasks.workunit.client.0.smithi067.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-04-01T04:51:34.269 INFO:tasks.workunit.client.0.smithi067.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 9c 49 45 d3 72 84 |re.........IE.r.| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000230 46 5a ad 54 3b 37 ad 97 50 eb 00 00 00 00 00 00 |FZ.T;7..P.......| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000240 00 00 1e 00 00 00 3b 3d 0a 66 29 bc 03 1c 01 00 |......;=.f).....| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:0000025e 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:34.270 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:0000003f 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:34.271 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:34.272 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:34.273 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:34.274 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-04-01T04:51:34.275 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:34.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-04-01T04:51:34.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-04-01T04:51:34.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-04-01T04:51:34.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-04-01T04:51:36.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-04-01T04:51:36.159 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-04-01T04:51:36.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-04-01T04:51:36.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:51:36.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-04-01T04:51:36.162 INFO:tasks.workunit.client.0.smithi067.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-04-01T04:51:36.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-04-01T04:51:36.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-04-01T04:51:36.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-04-01T04:51:36.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.185+0000 7f15ecc3e980 1 bdev(0x561b78a7a000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.185+0000 7f15ecc3e980 0 bdev(0x561b78a7a000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 1 bdev(0x561b78a7a000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 1 bdev(0x561b78a7a700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 0 bdev(0x561b78a7a700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 1 bdev(0x561b78a7a700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-04-01T04:51:36.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.186+0000 7f15ecc3e980 1 bluefs mount 2024-04-01T04:51:36.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.191+0000 7f15ecc3e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:36.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.202+0000 7f15ecc3e980 1 bluefs mount shared_bdev_used = 0 2024-04-01T04:51:36.203 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.203+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:36.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:36.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.205+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.206+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:36.207 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.206+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.210+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.210+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.210+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.210+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.210+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.211+0000 7f15ecc3e980 1 freelist init 2024-04-01T04:51:36.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.211+0000 7f15ecc3e980 1 freelist _read_cfg 2024-04-01T04:51:36.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.211+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 8.77383e-07 2024-04-01T04:51:36.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.211+0000 7f15ecc3e980 1 bluefs umount 2024-04-01T04:51:36.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.211+0000 7f15ecc3e980 1 bdev(0x561b78a7a700 td/osd-mapper/0/block) close 2024-04-01T04:51:36.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.448+0000 7f15ecc3e980 1 bdev(0x561b78a7a700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-04-01T04:51:36.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.448+0000 7f15ecc3e980 0 bdev(0x561b78a7a700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-04-01T04:51:36.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.448+0000 7f15ecc3e980 1 bdev(0x561b78a7a700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:36.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.448+0000 7f15ecc3e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-04-01T04:51:36.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.448+0000 7f15ecc3e980 1 bluefs mount 2024-04-01T04:51:36.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.458+0000 7f15ecc3e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:36.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.466+0000 7f15ecc3e980 1 bluefs mount shared_bdev_used = 9568256 2024-04-01T04:51:36.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.466+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.468+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.468+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.468+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.468+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:36.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:36.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.469+0000 7f15ecc3e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:36.721 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.720+0000 7f15ecc3e980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:36.723 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-04-01T04:51:36.723 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 0b 05 58 02 00 00 55 24 bd 08 99 fb 4d 64 82 f9 |..X...U$....Md..| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 36 77 87 50 08 f4 00 00 00 00 1e 00 00 00 00 00 |6w.P............| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:* 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-04-01T04:51:36.724 INFO:tasks.workunit.client.0.smithi067.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-04-01T04:51:36.725 INFO:tasks.workunit.client.0.smithi067.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 36 d7 76 67 40 1b |re........6.vg@.| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000230 4b 37 81 2b 93 c2 31 d6 9f 78 00 00 00 00 00 00 |K7.+..1..x......| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000240 00 00 1e 00 00 00 fa 3c 0a 66 98 c9 38 10 01 00 |.......<.f..8...| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:0000025e 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:36.726 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:0000003f 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:36.727 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.728 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:36.729 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-04-01T04:51:36.730 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.731 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-04-01T04:51:36.731 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:36.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.721+0000 7f15ecc3e980 1 bluefs umount 2024-04-01T04:51:36.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.722+0000 7f15ecc3e980 1 bdev(0x561b78a7a700 td/osd-mapper/0/block) close 2024-04-01T04:51:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.959+0000 7f15ecc3e980 1 freelist shutdown 2024-04-01T04:51:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:36.959+0000 7f15ecc3e980 1 bdev(0x561b78a7a000 td/osd-mapper/0/block) close 2024-04-01T04:51:37.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-04-01T04:51:37.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-04-01T04:51:37.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-04-01T04:51:38.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-04-01T04:51:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-04-01T04:51:38.245 INFO:tasks.workunit.client.0.smithi067.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:38.245 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-04-01T04:51:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.vbZyH0QYON_the_val 2024-04-01T04:51:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.vbZyH0QYON_the_val' 2024-04-01T04:51:38.247 INFO:tasks.workunit.client.0.smithi067.stdout:Value dumped in: /tmp/tmp.vbZyH0QYON_the_val 2024-04-01T04:51:38.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.vbZyH0QYON_the_val 2024-04-01T04:51:38.782 INFO:tasks.workunit.client.0.smithi067.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-04-01T04:51:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:39.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.vbZyH0QYON_the_val 2024-04-01T04:51:39.309 INFO:tasks.workunit.client.0.smithi067.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-04-01T04:51:39.309 INFO:tasks.workunit.client.0.smithi067.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-04-01T04:51:39.309 INFO:tasks.workunit.client.0.smithi067.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-04-01T04:51:39.309 INFO:tasks.workunit.client.0.smithi067.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stdout:0000073 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-04-01T04:51:39.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:40.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.vbZyH0QYON_the_val 2024-04-01T04:51:41.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:41.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:51:41.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-04-01T04:51:41.394 INFO:tasks.workunit.client.0.smithi067.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-04-01T04:51:41.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-04-01T04:51:41.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-04-01T04:51:41.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-04-01T04:51:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bdev(0x556ca7e5c000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-04-01T04:51:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 0 bdev(0x556ca7e5c000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-04-01T04:51:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bdev(0x556ca7e5c000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-04-01T04:51:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bdev(0x556ca7e5c700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-04-01T04:51:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 0 bdev(0x556ca7e5c700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-04-01T04:51:41.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bdev(0x556ca7e5c700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:41.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-04-01T04:51:41.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.417+0000 7fcabe640980 1 bluefs mount 2024-04-01T04:51:41.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.427+0000 7fcabe640980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:41.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.436+0000 7fcabe640980 1 bluefs mount shared_bdev_used = 0 2024-04-01T04:51:41.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.436+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.438+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.438+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:41.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:41.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:41.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:41.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.439+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 freelist init 2024-04-01T04:51:41.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.444+0000 7fcabe640980 1 freelist _read_cfg 2024-04-01T04:51:41.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.445+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 6.10353e-07 2024-04-01T04:51:41.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.445+0000 7fcabe640980 1 bluefs umount 2024-04-01T04:51:41.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.445+0000 7fcabe640980 1 bdev(0x556ca7e5c700 td/osd-mapper/1/block) close 2024-04-01T04:51:41.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.680+0000 7fcabe640980 1 bdev(0x556ca7e5c700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-04-01T04:51:41.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.680+0000 7fcabe640980 0 bdev(0x556ca7e5c700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-04-01T04:51:41.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.680+0000 7fcabe640980 1 bdev(0x556ca7e5c700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:41.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.680+0000 7fcabe640980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-04-01T04:51:41.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.680+0000 7fcabe640980 1 bluefs mount 2024-04-01T04:51:41.685 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.685+0000 7fcabe640980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:41.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.693+0000 7fcabe640980 1 bluefs mount shared_bdev_used = 9568256 2024-04-01T04:51:41.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.693+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:41.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.695+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:41.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:41.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:41.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.696+0000 7fcabe640980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:41.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.946+0000 7fcabe640980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 0b 05 58 02 00 00 55 24 bd 08 99 fb 4d 64 82 f9 |..X...U$....Md..| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 36 77 87 50 08 f4 01 00 00 00 1e 00 00 00 00 00 |6w.P............| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:* 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-04-01T04:51:41.948 INFO:tasks.workunit.client.0.smithi067.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-04-01T04:51:41.949 INFO:tasks.workunit.client.0.smithi067.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000220 72 65 1e 00 00 00 0a 00 00 00 7b bb a6 87 a5 68 |re........{....h| 2024-04-01T04:51:41.950 INFO:tasks.workunit.client.0.smithi067.stdout:00000230 4c d7 84 26 90 15 53 c6 d1 1b 00 00 00 00 00 00 |L..&..S.........| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000240 00 00 1e 00 00 00 1a 3d 0a 66 bd b9 f4 1a 01 00 |.......=.f......| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:0000025e 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:41.951 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:0000003f 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:41.952 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:41.953 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:41.954 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-04-01T04:51:41.955 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:41.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.946+0000 7fcabe640980 1 bluefs umount 2024-04-01T04:51:41.956 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:41.946+0000 7fcabe640980 1 bdev(0x556ca7e5c700 td/osd-mapper/1/block) close 2024-04-01T04:51:42.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:42.183+0000 7fcabe640980 1 freelist shutdown 2024-04-01T04:51:42.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:42.183+0000 7fcabe640980 1 bdev(0x556ca7e5c000 td/osd-mapper/1/block) close 2024-04-01T04:51:42.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-04-01T04:51:42.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-04-01T04:51:42.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-04-01T04:51:43.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:43.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:43.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-04-01T04:51:43.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-04-01T04:51:43.233 INFO:tasks.workunit.client.0.smithi067.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:43.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-04-01T04:51:43.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.9zVKahHYs9_the_val 2024-04-01T04:51:43.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:43.235 INFO:tasks.workunit.client.0.smithi067.stdout:Value dumped in: /tmp/tmp.9zVKahHYs9_the_val 2024-04-01T04:51:43.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.9zVKahHYs9_the_val' 2024-04-01T04:51:43.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.9zVKahHYs9_the_val 2024-04-01T04:51:43.840 INFO:tasks.workunit.client.0.smithi067.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-04-01T04:51:44.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.9zVKahHYs9_the_val 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-04-01T04:51:44.280 INFO:tasks.workunit.client.0.smithi067.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-04-01T04:51:44.281 INFO:tasks.workunit.client.0.smithi067.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-04-01T04:51:44.281 INFO:tasks.workunit.client.0.smithi067.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-04-01T04:51:44.281 INFO:tasks.workunit.client.0.smithi067.stdout:0000073 2024-04-01T04:51:44.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-04-01T04:51:44.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:45.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.9zVKahHYs9_the_val 2024-04-01T04:51:46.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:46.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:51:46.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-04-01T04:51:46.135 INFO:tasks.workunit.client.0.smithi067.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-04-01T04:51:46.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-04-01T04:51:46.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-04-01T04:51:46.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bdev(0x55faf2ff6000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 0 bdev(0x55faf2ff6000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bdev(0x55faf2ff6000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bdev(0x55faf2ff6700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 0 bdev(0x55faf2ff6700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-04-01T04:51:46.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bdev(0x55faf2ff6700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:46.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-04-01T04:51:46.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.158+0000 7fb65aa48980 1 bluefs mount 2024-04-01T04:51:46.168 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.167+0000 7fb65aa48980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:46.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.176+0000 7fb65aa48980 1 bluefs mount shared_bdev_used = 0 2024-04-01T04:51:46.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.176+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.178+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.178+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.178+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.178+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.179+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:46.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 freelist init 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.184+0000 7fb65aa48980 1 freelist _read_cfg 2024-04-01T04:51:46.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.185+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 5.34059e-07 2024-04-01T04:51:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.185+0000 7fb65aa48980 1 bluefs umount 2024-04-01T04:51:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.185+0000 7fb65aa48980 1 bdev(0x55faf2ff6700 td/osd-mapper/2/block) close 2024-04-01T04:51:46.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.424+0000 7fb65aa48980 1 bdev(0x55faf2ff6700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-04-01T04:51:46.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.424+0000 7fb65aa48980 0 bdev(0x55faf2ff6700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-04-01T04:51:46.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.424+0000 7fb65aa48980 1 bdev(0x55faf2ff6700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-04-01T04:51:46.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.424+0000 7fb65aa48980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-04-01T04:51:46.425 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.424+0000 7fb65aa48980 1 bluefs mount 2024-04-01T04:51:46.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.434+0000 7fb65aa48980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-04-01T04:51:46.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.442+0000 7fb65aa48980 1 bluefs mount shared_bdev_used = 9568256 2024-04-01T04:51:46.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.442+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.444+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.444+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.444+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.444+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.444+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-04-01T04:51:46.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-04-01T04:51:46.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-04-01T04:51:46.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-04-01T04:51:46.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-04-01T04:51:46.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.445+0000 7fb65aa48980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-04-01T04:51:46.687 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.686+0000 7fb65aa48980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 0b 05 58 02 00 00 55 24 bd 08 99 fb 4d 64 82 f9 |..X...U$....Md..| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 36 77 87 50 08 f4 02 00 00 00 1e 00 00 00 00 00 |6w.P............| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:* 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-04-01T04:51:46.689 INFO:tasks.workunit.client.0.smithi067.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-04-01T04:51:46.690 INFO:tasks.workunit.client.0.smithi067.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 9c 49 45 d3 72 84 |re.........IE.r.| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000230 46 5a ad 54 3b 37 ad 97 50 eb 00 00 00 00 00 00 |FZ.T;7..P.......| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000240 00 00 1e 00 00 00 3b 3d 0a 66 29 bc 03 1c 01 00 |......;=.f).....| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout:0000025e 2024-04-01T04:51:46.691 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:46.692 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:0000003f 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000047 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:46.693 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:46.694 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:46.695 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:0000003b 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-04-01T04:51:46.696 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:51:46.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.687+0000 7fb65aa48980 1 bluefs umount 2024-04-01T04:51:46.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.687+0000 7fb65aa48980 1 bdev(0x55faf2ff6700 td/osd-mapper/2/block) close 2024-04-01T04:51:46.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.927+0000 7fb65aa48980 1 freelist shutdown 2024-04-01T04:51:46.927 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:46.927+0000 7fb65aa48980 1 bdev(0x55faf2ff6000 td/osd-mapper/2/block) close 2024-04-01T04:51:47.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-04-01T04:51:47.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-04-01T04:51:47.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-04-01T04:51:48.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:48.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:48.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-04-01T04:51:48.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-04-01T04:51:48.219 INFO:tasks.workunit.client.0.smithi067.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:48.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-04-01T04:51:48.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.cquTIt2CYB_the_val 2024-04-01T04:51:48.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:48.221 INFO:tasks.workunit.client.0.smithi067.stdout:Value dumped in: /tmp/tmp.cquTIt2CYB_the_val 2024-04-01T04:51:48.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.cquTIt2CYB_the_val' 2024-04-01T04:51:48.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.cquTIt2CYB_the_val 2024-04-01T04:51:48.714 INFO:tasks.workunit.client.0.smithi067.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-04-01T04:51:49.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:49.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.cquTIt2CYB_the_val 2024-04-01T04:51:49.017 INFO:tasks.workunit.client.0.smithi067.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-04-01T04:51:49.017 INFO:tasks.workunit.client.0.smithi067.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-04-01T04:51:49.017 INFO:tasks.workunit.client.0.smithi067.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-04-01T04:51:49.017 INFO:tasks.workunit.client.0.smithi067.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-04-01T04:51:49.018 INFO:tasks.workunit.client.0.smithi067.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-04-01T04:51:49.018 INFO:tasks.workunit.client.0.smithi067.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-04-01T04:51:49.018 INFO:tasks.workunit.client.0.smithi067.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-04-01T04:51:49.018 INFO:tasks.workunit.client.0.smithi067.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-04-01T04:51:49.019 INFO:tasks.workunit.client.0.smithi067.stdout:0000073 2024-04-01T04:51:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-04-01T04:51:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:51:49.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.cquTIt2CYB_the_val 2024-04-01T04:51:50.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:51:50.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-04-01T04:51:50.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-04-01T04:51:50.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:50.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:51:50.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-04-01T04:51:50.856 INFO:tasks.workunit.client.0.smithi067.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-04-01T04:51:50.856 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-04-01T04:51:50.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:51:50.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28982/$cluster-$name.asok' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28982/$cluster-$name.asok' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-04-01T04:51:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T04:51:50.864 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T04:51:50.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28982/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T04:51:50.864 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-04-01T04:51:50.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T04:51:50.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T04:51:50.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T04:51:50.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T04:51:50.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:50.900+0000 7f6b3fcd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:50.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:50.902+0000 7f6b3fcd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:50.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:50.903+0000 7f6b3fcd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:51.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T04:51:51.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:51:51.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T04:51:51.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:51:51.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:51:51.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:51.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:51:51.167 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:51:51.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:51.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:51:51.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:51.418+0000 7f6b3fcd3640 -1 Falling back to public interface 2024-04-01T04:51:51.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:51:52.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:51:52.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:52.501 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:51:52.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:51:52.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:52.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:51:52.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:51:53.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:53.745+0000 7f6b3fcd3640 -1 osd.0 30 log_to_monitors true 2024-04-01T04:51:53.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:51:53.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:53.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:51:53.807 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:51:53.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:53.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:51:54.144 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 33 up_thru 18 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6802/2183902611,v1:127.0.0.1:6803/2183902611] [v2:127.0.0.1:6804/2183902611,v1:127.0.0.1:6805/2183902611] exists,up 36d77667-401b-4b37-812b-93c231d69f78 2024-04-01T04:51:54.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:51:54.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:51:54.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-04-01T04:51:54.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-04-01T04:51:54.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T04:51:54.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T04:51:54.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28982/$cluster-$name.asok' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28982/$cluster-$name.asok' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:51:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:51:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:51:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:51:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T04:51:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T04:51:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-04-01T04:51:54.151 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28982/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T04:51:54.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T04:51:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:54.183+0000 7f89c9fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:54.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:54.218+0000 7f89c9fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:54.219+0000 7f89c9fbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:54.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:51:54.516 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:51:54.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:54.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:51:54.745 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:54.745+0000 7f89c9fbd640 -1 Falling back to public interface 2024-04-01T04:51:54.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:51:55.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:51:55.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:55.869 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:51:55.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:51:55.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:55.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:51:56.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:51:57.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:57.013+0000 7f89c9fbd640 -1 osd.1 30 log_to_monitors true 2024-04-01T04:51:57.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:51:57.177 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:51:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:51:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:57.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:51:57.663 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 36 up_thru 18 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/721790017,v1:127.0.0.1:6811/721790017] [v2:127.0.0.1:6812/721790017,v1:127.0.0.1:6813/721790017] exists,up 7bbba687-a568-4cd7-8426-901553c6d11b 2024-04-01T04:51:57.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:51:57.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:51:57.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:51:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28982/$cluster-$name.asok' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28982/$cluster-$name.asok' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-04-01T04:51:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T04:51:57.670 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T04:51:57.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5524bd08-99fb-4d64-82f9-3677875008f4 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28982/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T04:51:57.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-04-01T04:51:57.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T04:51:57.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T04:51:57.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T04:51:57.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T04:51:57.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:57.702+0000 7f5a9d699640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:57.795 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:57.794+0000 7f5a9d699640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:57.796 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:57.795+0000 7f5a9d699640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:51:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T04:51:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:51:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T04:51:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:51:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:51:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:58.002 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:51:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:51:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:51:58.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:51:58.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:51:58.313+0000 7f5a9d699640 -1 Falling back to public interface 2024-04-01T04:51:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:51:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:51:59.295 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:51:59.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:51:59.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:51:59.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:51:59.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:00.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:00.536+0000 7f5a9d699640 -1 osd.2 30 log_to_monitors true 2024-04-01T04:52:00.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:00.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:00.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:52:00.598 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:52:00.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:52:00.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:01.940 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:52:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:52:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:52:02.299 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 39 up_thru 39 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/2844196121,v1:127.0.0.1:6819/2844196121] [v2:127.0.0.1:6820/2844196121,v1:127.0.0.1:6821/2844196121] exists,up 9c4945d3-7284-465a-ad54-3b37ad9750eb 2024-04-01T04:52:02.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:52:02.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:52:02.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:52:02.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-04-01T04:52:03.302 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-04-01T04:52:03.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-04-01T04:52:03.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:52:03.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-04-01T04:52:03.413 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:52:03.413 INFO:tasks.workunit.client.0.smithi067.stdout: "version": "19.0.0-1977-g384f31f3", 2024-04-01T04:52:03.413 INFO:tasks.workunit.client.0.smithi067.stdout: "release": "squid", 2024-04-01T04:52:03.413 INFO:tasks.workunit.client.0.smithi067.stdout: "release_type": "dev" 2024-04-01T04:52:03.413 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:52:03.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:52:03.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-04-01T04:52:03.522 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:52:03.522 INFO:tasks.workunit.client.0.smithi067.stdout: "version": "19.0.0-1977-g384f31f3", 2024-04-01T04:52:03.523 INFO:tasks.workunit.client.0.smithi067.stdout: "release": "squid", 2024-04-01T04:52:03.523 INFO:tasks.workunit.client.0.smithi067.stdout: "release_type": "dev" 2024-04-01T04:52:03.523 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:52:03.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-04-01T04:52:03.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-04-01T04:52:03.652 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:52:03.652 INFO:tasks.workunit.client.0.smithi067.stdout: "version": "19.0.0-1977-g384f31f3", 2024-04-01T04:52:03.653 INFO:tasks.workunit.client.0.smithi067.stdout: "release": "squid", 2024-04-01T04:52:03.653 INFO:tasks.workunit.client.0.smithi067.stdout: "release_type": "dev" 2024-04-01T04:52:03.653 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:52:03.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-04-01T04:52:03.688 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "objxxx", 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "clones": [ 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [ 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 1, 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap01" 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.689 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 2, 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap02" 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 6, 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "overlaps": [] 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [ 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 3, 2024-04-01T04:52:03.690 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap13" 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 5, 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "overlaps": [] 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [ 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 4, 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap24" 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "id": 5, 2024-04-01T04:52:03.691 INFO:tasks.workunit.client.0.smithi067.stdout: "name": "snap25" 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 6, 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: "overlaps": [] 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: { 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: "id": "head", 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: "snapshots": [], 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: "size": 6 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: } 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout: ] 2024-04-01T04:52:03.692 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:52:03.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-04-01T04:52:04.071 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is unset 2024-04-01T04:52:04.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-04-01T04:52:04.556 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is unset 2024-04-01T04:52:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-04-01T04:52:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-04-01T04:52:04.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-04-01T04:52:04.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 activating 2024-04-01T04:52:02.125770+0000 28'7 41:60 [1,2,0] 1 [1,2,0] 1 28'7 2024-04-01T04:51:21.139526+0000 28'7 2024-04-01T04:51:21.139526+0000 0 0 no scrub is scheduled 4 0 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:01.866855+0000 0'0 41:43 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-04-01T04:51:57.962530+0000 0'0 37:17 [0,1] 0 [0,1] 0 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 activating 2024-04-01T04:52:02.125811+0000 0'0 41:28 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:52:05.115 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:52:05.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-04-01T04:52:07.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-04-01T04:52:07.232 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:52:07.232 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": true, 2024-04-01T04:52:07.232 INFO:tasks.workunit.client.0.smithi067.stdout: "must": true, 2024-04-01T04:52:07.232 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "0.000000" 2024-04-01T04:52:07.233 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:52:07.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-04-01T04:52:12.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-04-01T04:52:02.360874+0000 28'7 43:72 [1,2,0] 1 [1,2,0] 1 28'7 2024-04-01T04:51:21.139526+0000 28'7 2024-04-01T04:51:21.139526+0000 0 0 queued for deep scrub 4 0 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:01.866855+0000 0'0 43:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:02.294146+0000 0'0 43:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:02.294139+0000 0'0 43:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:52:12.521 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:52:12.522 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:52:12.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:52:12.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-04-01T04:52:12.533 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:52:08.110+0000 7f89ac733640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-04-01T04:52:12.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-04-01T04:52:12.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-04-01T04:52:12.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-04-01T04:52:12.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-04-01T04:52:12.537 INFO:tasks.workunit.client.0.smithi067.stdout:prev count: 1 2024-04-01T04:52:12.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-04-01T04:52:12.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-04-01T04:52:12.632 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:52:12.632 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": true, 2024-04-01T04:52:12.632 INFO:tasks.workunit.client.0.smithi067.stdout: "must": true, 2024-04-01T04:52:12.632 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "0.000000" 2024-04-01T04:52:12.632 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:52:12.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-04-01T04:52:17.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-04-01T04:52:17.917 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:52:17.917 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-04-01T04:52:08.112373+0000 28'7 43:80 [1,2,0] 1 [1,2,0] 1 28'7 2024-04-01T04:52:08.112273+0000 28'7 2024-04-01T04:52:08.112273+0000 0 1 periodic scrub scheduled @ 2024-04-02T04:52:08.112273+0000 4 0 2024-04-01T04:52:17.917 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:01.866855+0000 0'0 43:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:17.918 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:02.294146+0000 0'0 43:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:17.918 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:52:02.294139+0000 0'0 43:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-01T04:51:09.062383+0000 0'0 2024-04-01T04:51:09.062383+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:51:09.062383+0000 0 0 2024-04-01T04:52:17.918 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:52:17.918 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:52:17.918 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:52:17.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:52:17.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-04-01T04:52:17.930 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:52:08.110+0000 7f89ac733640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-04-01T04:52:17.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-04-01T04:52:17.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-04-01T04:52:17.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-04-01T04:52:17.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-04-01T04:52:17.933 INFO:tasks.workunit.client.0.smithi067.stdout:current count: 1 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:52:17.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:52:18.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:52:18.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-04-01T04:52:18.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-04-01T04:52:18.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-04-01T04:52:18.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-04-01T04:52:18.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-04-01T04:52:19.776 INFO:tasks.workunit.client.0.smithi067.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:52:19.776 INFO:tasks.workunit.client.0.smithi067.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-04-01T04:52:19.777 INFO:tasks.workunit.client.0.smithi067.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-04-01T04:52:19.777 INFO:tasks.workunit.client.0.smithi067.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-04-01T04:52:19.777 INFO:tasks.workunit.client.0.smithi067.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:52:19.777 INFO:tasks.workunit.client.0.smithi067.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-04-01T04:52:19.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-04-01T04:52:19.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-04-01T04:52:19.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-04-01T04:52:19.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-04-01T04:52:20.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-04-01T04:52:20.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-04-01T04:52:20.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-04-01T04:52:20.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:52:20.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:52:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:52:20.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:52:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:52:20.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:52:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:52:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:52:20.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:52:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:52:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:52:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:52:20.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:52:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:52:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:52:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-04-01T04:52:20.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:52:20.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:20.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:52:20.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28982 2024-04-01T04:52:20.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:52:20.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:52:20.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:52:20.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:52:20.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:52:20.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:52:20.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:52:20.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:52:20.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:52:20.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:52:20.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:52:20.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:52:20.962 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28982 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28982 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:52:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-04-01T04:52:20.969 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-01T04:52:20.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-01T04:52:21.039 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-04-01T04:52:21.040 DEBUG:teuthology.orchestra.run.smithi067:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-04-01T04:52:21.102 INFO:tasks.workunit.client.0.smithi067.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-01T04:52:21.108 INFO:tasks.workunit.client.0.smithi067.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-01T04:52:21.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:52:21.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:52:21.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-01T04:52:21.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-01T04:52:21.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none ' 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-04-01T04:52:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-04-01T04:52:21.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-04-01T04:52:21.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_recovery_scrub_2' 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.45195 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-04-01T04:52:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:52:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:52:21.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:52:21.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:52:21.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:52:21.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:52:21.115 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:52:21.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:52:21.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:52:21.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-04-01T04:52:21.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:52:21.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:21.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:21.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45195 2024-04-01T04:52:21.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:52:21.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:52:21.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-04-01T04:52:21.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T04:52:21.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:21.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:21.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.45195 2024-04-01T04:52:21.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T04:52:21.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-04-01T04:52:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-01T04:52:21.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T04:52:21.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:52:21.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:52:21.509 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:52:21.509 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:21.510 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:21.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:21.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-01T04:52:21.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T04:52:21.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T04:52:21.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T04:52:21.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T04:52:21.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T04:52:21.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T04:52:21.706 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T04:52:21.706 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T04:52:21.707 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T04:52:21.707 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:52:21.707 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:21.707 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:21.707 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45195/ceph-mon.a.asok 2024-04-01T04:52:21.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T04:52:21.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45195/ceph-mon.a.asok config get fsid 2024-04-01T04:52:21.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T04:52:21.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T04:52:21.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T04:52:21.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T04:52:21.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45195/ceph-mon.a.asok 2024-04-01T04:52:21.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T04:52:21.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45195/ceph-mon.a.asok config get mon_host 2024-04-01T04:52:21.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-04-01T04:52:21.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-04-01T04:52:21.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T04:52:21.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T04:52:21.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T04:52:21.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-04-01T04:52:21.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T04:52:22.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T04:52:22.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T04:52:22.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T04:52:22.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-04-01T04:52:22.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-04-01T04:52:22.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-04-01T04:52:22.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-04-01T04:52:22.714 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-04-01T04:52:22.714 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-04-01T04:52:22.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:52:22.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-04-01T04:52:22.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:52:22.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:52:22.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T04:52:22.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:22.720 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-04-01T04:52:22.721 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:52:22.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dbaeb3bc-5cd8-429b-9a8c-68266ce6038f 2024-04-01T04:52:22.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dbaeb3bc-5cd8-429b-9a8c-68266ce6038f' 2024-04-01T04:52:22.722 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 dbaeb3bc-5cd8-429b-9a8c-68266ce6038f 2024-04-01T04:52:22.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:52:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGPQpmblXXKxAAHkhGIWoXChBFR6E85/Baxg== 2024-04-01T04:52:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGPQpmblXXKxAAHkhGIWoXChBFR6E85/Baxg=="}' 2024-04-01T04:52:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dbaeb3bc-5cd8-429b-9a8c-68266ce6038f -i td/osd-recovery-scrub/0/new.json 2024-04-01T04:52:22.942 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:52:22.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-04-01T04:52:22.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCGPQpmblXXKxAAHkhGIWoXChBFR6E85/Baxg== --osd-uuid dbaeb3bc-5cd8-429b-9a8c-68266ce6038f 2024-04-01T04:52:22.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:22.982+0000 7f17fbd38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:22.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:22.993+0000 7f17fbd38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:22.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:22.994+0000 7f17fbd38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:22.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:22.994+0000 7f17fbd38640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-04-01T04:52:22.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:22.994+0000 7f17fbd38640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-04-01T04:52:22.995 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:22.994+0000 7f17fbd38640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-04-01T04:52:23.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:23.085+0000 7f17fbd38640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-04-01T04:52:28.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-04-01T04:52:28.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:52:28.713 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T04:52:28.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T04:52:28.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:52:29.091 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T04:52:29.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T04:52:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-04-01T04:52:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:52:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:52:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:52:29.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:29.117+0000 7f5214319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:29.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:29.119+0000 7f5214319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:29.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:29.120+0000 7f5214319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:29.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T04:52:29.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:52:29.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T04:52:29.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:52:29.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:52:29.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:29.375 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:52:29.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:52:29.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:29.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:52:29.648 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:29.648+0000 7f5214319640 -1 Falling back to public interface 2024-04-01T04:52:29.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:30.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:30.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:30.654 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:52:30.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:52:30.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:30.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:52:30.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:31.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:31.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:31.946 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:52:31.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:52:31.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:31.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:52:31.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:31.971+0000 7f5214319640 -1 osd.0 0 log_to_monitors true 2024-04-01T04:52:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:33.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:33.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:33.305 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:52:33.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:52:33.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:33.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:52:33.600 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/283360876,v1:127.0.0.1:6803/283360876] [v2:127.0.0.1:6804/283360876,v1:127.0.0.1:6805/283360876] exists,up dbaeb3bc-5cd8-429b-9a8c-68266ce6038f 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:52:33.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:52:33.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:52:33.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6cd195b-7945-4ee9-9c3d-3bd097bcf220 2024-04-01T04:52:33.607 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 d6cd195b-7945-4ee9-9c3d-3bd097bcf220 2024-04-01T04:52:33.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d6cd195b-7945-4ee9-9c3d-3bd097bcf220' 2024-04-01T04:52:33.608 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:52:33.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRPQpmcQD1JBAAA59JyT1jylqAW0zRHSI5YQ== 2024-04-01T04:52:33.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRPQpmcQD1JBAAA59JyT1jylqAW0zRHSI5YQ=="}' 2024-04-01T04:52:33.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6cd195b-7945-4ee9-9c3d-3bd097bcf220 -i td/osd-recovery-scrub/1/new.json 2024-04-01T04:52:33.974 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:52:33.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-04-01T04:52:33.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCRPQpmcQD1JBAAA59JyT1jylqAW0zRHSI5YQ== --osd-uuid d6cd195b-7945-4ee9-9c3d-3bd097bcf220 2024-04-01T04:52:34.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.013+0000 7f5e49c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:34.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.050+0000 7f5e49c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:34.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.051+0000 7f5e49c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:34.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.051+0000 7f5e49c13640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-04-01T04:52:34.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.051+0000 7f5e49c13640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-04-01T04:52:34.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.051+0000 7f5e49c13640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-04-01T04:52:34.125 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:34.125+0000 7f5e49c13640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-04-01T04:52:39.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-04-01T04:52:39.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:52:39.870 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T04:52:39.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T04:52:39.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:52:40.236 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T04:52:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T04:52:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-04-01T04:52:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:52:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:52:40.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:52:40.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:40.263+0000 7f5d90a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:40.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:40.264+0000 7f5d90a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:40.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:40.265+0000 7f5d90a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:40.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T04:52:40.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:52:40.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T04:52:40.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:52:40.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:52:40.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:40.523 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:52:40.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:52:40.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:40.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:52:40.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:40.790+0000 7f5d90a0c640 -1 Falling back to public interface 2024-04-01T04:52:40.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:41.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:41.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:41.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:52:41.805 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:52:41.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:41.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:52:42.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:43.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:43.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:43.097 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:52:43.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:52:43.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:43.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:52:43.173 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:43.172+0000 7f5d90a0c640 -1 osd.1 0 log_to_monitors true 2024-04-01T04:52:43.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:44.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:44.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:44.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:52:44.391 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:52:44.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:44.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:52:44.681 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1551512299,v1:127.0.0.1:6811/1551512299] [v2:127.0.0.1:6812/1551512299,v1:127.0.0.1:6813/1551512299] exists,up d6cd195b-7945-4ee9-9c3d-3bd097bcf220 2024-04-01T04:52:44.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:52:44.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:52:44.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:52:44.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:52:44.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-04-01T04:52:44.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:52:44.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:52:44.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=06d08f89-59f5-40e3-95a9-81f190b10683 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 06d08f89-59f5-40e3-95a9-81f190b10683' 2024-04-01T04:52:44.688 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 06d08f89-59f5-40e3-95a9-81f190b10683 2024-04-01T04:52:44.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:52:44.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcPQpmW2rPKRAAvjkgfa+fKBN+0zfgF5SQNg== 2024-04-01T04:52:44.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcPQpmW2rPKRAAvjkgfa+fKBN+0zfgF5SQNg=="}' 2024-04-01T04:52:44.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 06d08f89-59f5-40e3-95a9-81f190b10683 -i td/osd-recovery-scrub/2/new.json 2024-04-01T04:52:45.068 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:52:45.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-04-01T04:52:45.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCcPQpmW2rPKRAAvjkgfa+fKBN+0zfgF5SQNg== --osd-uuid 06d08f89-59f5-40e3-95a9-81f190b10683 2024-04-01T04:52:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.106+0000 7f47a9119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:45.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.143+0000 7f47a9119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:45.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.144+0000 7f47a9119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:45.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.144+0000 7f47a9119640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-04-01T04:52:45.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.144+0000 7f47a9119640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-04-01T04:52:45.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.144+0000 7f47a9119640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-04-01T04:52:45.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:45.209+0000 7f47a9119640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-04-01T04:52:50.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-04-01T04:52:50.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:52:50.934 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T04:52:50.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T04:52:50.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:52:51.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T04:52:51.301 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T04:52:51.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-04-01T04:52:51.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:52:51.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:52:51.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:52:51.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:51.326+0000 7f55a26c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:51.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:51.328+0000 7f55a26c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:51.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:51.329+0000 7f55a26c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:51.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T04:52:51.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:52:51.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T04:52:51.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:52:51.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:52:51.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:51.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:52:51.586 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:52:51.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:51.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:52:51.846 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:51.845+0000 7f55a26c8640 -1 Falling back to public interface 2024-04-01T04:52:51.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:52.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:52.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:52.891 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:52:52.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:52:52.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:52.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:52:53.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:54.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:54.182 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:52:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:52:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:54.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:52:54.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:54.292+0000 7f55a26c8640 -1 osd.2 0 log_to_monitors true 2024-04-01T04:52:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:52:55.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:52:55.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:52:55.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:52:55.503 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:52:55.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:52:55.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:52:55.830 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2262312784,v1:127.0.0.1:6819/2262312784] [v2:127.0.0.1:6820/2262312784,v1:127.0.0.1:6821/2262312784] exists,up 06d08f89-59f5-40e3-95a9-81f190b10683 2024-04-01T04:52:55.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:52:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:52:55.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:52:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-04-01T04:52:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-04-01T04:52:55.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:52:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1803cece-1894-4705-b3b0-de4e0a57dcbf 2024-04-01T04:52:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1803cece-1894-4705-b3b0-de4e0a57dcbf' 2024-04-01T04:52:55.838 INFO:tasks.workunit.client.0.smithi067.stdout:add osd3 1803cece-1894-4705-b3b0-de4e0a57dcbf 2024-04-01T04:52:55.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:52:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnPQpmPQOGMhAA2E/4+YFa0xov/G6BGxu8SQ== 2024-04-01T04:52:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnPQpmPQOGMhAA2E/4+YFa0xov/G6BGxu8SQ=="}' 2024-04-01T04:52:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1803cece-1894-4705-b3b0-de4e0a57dcbf -i td/osd-recovery-scrub/3/new.json 2024-04-01T04:52:56.206 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:52:56.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-04-01T04:52:56.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCnPQpmPQOGMhAA2E/4+YFa0xov/G6BGxu8SQ== --osd-uuid 1803cece-1894-4705-b3b0-de4e0a57dcbf 2024-04-01T04:52:56.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.246+0000 7fbbc199a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:56.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.291+0000 7fbbc199a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:56.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.292+0000 7fbbc199a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:52:56.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.292+0000 7fbbc199a640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-04-01T04:52:56.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.292+0000 7fbbc199a640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-04-01T04:52:56.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.292+0000 7fbbc199a640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-04-01T04:52:56.358 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:52:56.357+0000 7fbbc199a640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-04-01T04:53:02.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-04-01T04:53:02.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:53:02.314 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd3 key to auth repository 2024-04-01T04:53:02.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-01T04:53:02.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:53:02.684 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.3 2024-04-01T04:53:02.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-01T04:53:02.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-04-01T04:53:02.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:53:02.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:53:02.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:53:02.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:53:02.710+0000 7f6ce229d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:53:02.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:53:02.712+0000 7f6ce229d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:53:02.713 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:53:02.713+0000 7f6ce229d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:53:02.966 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:53:02.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:53:02.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:53:02.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:53:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:53:03.238+0000 7f6ce229d640 -1 Falling back to public interface 2024-04-01T04:53:03.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:53:04.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:53:04.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:53:04.253 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:53:04.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:53:04.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:53:04.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:53:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:53:05.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:53:05.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:53:05.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:53:05.551 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:53:05.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:53:05.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:53:05.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:53:05.576+0000 7f6ce229d640 -1 osd.3 0 log_to_monitors true 2024-04-01T04:53:05.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:53:06.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:53:06.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:53:06.885 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:53:06.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:53:06.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:53:06.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:53:07.177 INFO:tasks.workunit.client.0.smithi067.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2542056184,v1:127.0.0.1:6827/2542056184] [v2:127.0.0.1:6828/2542056184,v1:127.0.0.1:6829/2542056184] exists,up 1803cece-1894-4705-b3b0-de4e0a57dcbf 2024-04-01T04:53:07.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:53:07.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:53:07.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:53:07.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-04-01T04:53:07.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-01T04:53:07.555 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'test' created 2024-04-01T04:53:07.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T04:53:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-04-01T04:53:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T04:53:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T04:53:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T04:53:08.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T04:53:08.568 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T04:53:08.568 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T04:53:08.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T04:53:08.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T04:53:08.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T04:53:08.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T04:53:08.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T04:53:08.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T04:53:08.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T04:53:08.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T04:53:08.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:53:08.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T04:53:09.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-04-01T04:53:09.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-04-01T04:53:09.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-04-01T04:53:09.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:53:09.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T04:53:09.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-04-01T04:53:09.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-04-01T04:53:09.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-04-01T04:53:09.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:53:09.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T04:53:09.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-04-01T04:53:09.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-04-01T04:53:09.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2024-04-01T04:53:09.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:53:09.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T04:53:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-04-01T04:53:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-04-01T04:53:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444 3-85899345922' 2024-04-01T04:53:09.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:53:09.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-04-01T04:53:09.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:53:09.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T04:53:09.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-04-01T04:53:09.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:53:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-04-01T04:53:09.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-04-01T04:53:09.400 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836489 2024-04-01T04:53:09.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T04:53:09.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-04-01T04:53:09.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T04:53:10.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T04:53:10.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T04:53:10.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-04-01T04:53:10.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:53:10.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-04-01T04:53:10.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:53:10.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T04:53:10.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-04-01T04:53:10.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:53:10.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-04-01T04:53:10.982 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672966 2024-04-01T04:53:10.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-04-01T04:53:10.982 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T04:53:11.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-04-01T04:53:11.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:53:11.263 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-04-01T04:53:11.263 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:53:11.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T04:53:11.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-04-01T04:53:11.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:53:11.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-04-01T04:53:11.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-04-01T04:53:11.265 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509444 2024-04-01T04:53:11.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T04:53:11.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-04-01T04:53:11.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:53:11.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-04-01T04:53:11.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:53:11.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T04:53:11.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-04-01T04:53:11.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:53:11.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-04-01T04:53:11.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-04-01T04:53:11.667 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345922 2024-04-01T04:53:11.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T04:53:11.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-04-01T04:53:11.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T04:53:11.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T04:53:11.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T04:53:12.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T04:53:12.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T04:53:12.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T04:53:12.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T04:53:12.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T04:53:12.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T04:53:12.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T04:53:12.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T04:53:12.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-04-01T04:53:12.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-04-01T04:53:12.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-04-01T04:53:13.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-04-01T04:53:13.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-04-01T04:53:13.527 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:53:13.527 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:53:08.234143+0000 0'0 24:11 [1] 1 [1] 1 0'0 2024-04-01T04:53:07.503169+0000 0'0 2024-04-01T04:53:07.503169+0000 0 0 periodic scrub scheduled @ 2024-04-02T12:26:26.224172+0000 0 0 2024-04-01T04:53:13.527 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:53:13.527 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:53:13.527 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:53:13.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.45195 bs=1M count=50 2024-04-01T04:53:13.676 INFO:tasks.workunit.client.0.smithi067.stderr:50+0 records in 2024-04-01T04:53:13.676 INFO:tasks.workunit.client.0.smithi067.stderr:50+0 records out 2024-04-01T04:53:13.676 INFO:tasks.workunit.client.0.smithi067.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.137582 s, 381 MB/s 2024-04-01T04:53:13.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-04-01T04:53:13.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:13.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.45195 2024-04-01T04:53:15.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:15.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.45195 2024-04-01T04:53:16.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:16.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.45195 2024-04-01T04:53:18.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:18.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.45195 2024-04-01T04:53:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:19.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.45195 2024-04-01T04:53:21.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:21.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.45195 2024-04-01T04:53:22.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:22.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.45195 2024-04-01T04:53:23.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:23.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.45195 2024-04-01T04:53:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:25.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.45195 2024-04-01T04:53:26.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:26.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.45195 2024-04-01T04:53:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.45195 2024-04-01T04:53:29.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:29.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.45195 2024-04-01T04:53:31.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:31.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.45195 2024-04-01T04:53:32.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:32.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.45195 2024-04-01T04:53:34.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:34.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.45195 2024-04-01T04:53:35.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:35.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.45195 2024-04-01T04:53:37.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:37.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.45195 2024-04-01T04:53:38.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:38.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.45195 2024-04-01T04:53:40.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:40.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.45195 2024-04-01T04:53:41.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:41.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.45195 2024-04-01T04:53:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:43.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.45195 2024-04-01T04:53:44.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:44.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.45195 2024-04-01T04:53:46.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:46.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.45195 2024-04-01T04:53:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:48.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.45195 2024-04-01T04:53:49.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:49.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.45195 2024-04-01T04:53:51.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:51.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.45195 2024-04-01T04:53:52.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:52.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.45195 2024-04-01T04:53:54.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:54.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.45195 2024-04-01T04:53:55.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:55.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.45195 2024-04-01T04:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.45195 2024-04-01T04:53:58.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:53:58.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.45195 2024-04-01T04:54:00.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:00.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.45195 2024-04-01T04:54:01.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:01.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.45195 2024-04-01T04:54:03.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.45195 2024-04-01T04:54:04.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:04.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.45195 2024-04-01T04:54:06.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:06.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.45195 2024-04-01T04:54:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:07.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.45195 2024-04-01T04:54:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.45195 2024-04-01T04:54:10.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:10.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.45195 2024-04-01T04:54:12.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:12.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.45195 2024-04-01T04:54:14.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:14.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.45195 2024-04-01T04:54:15.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:15.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.45195 2024-04-01T04:54:17.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:17.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.45195 2024-04-01T04:54:18.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:18.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.45195 2024-04-01T04:54:20.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:20.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.45195 2024-04-01T04:54:21.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:21.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.45195 2024-04-01T04:54:23.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:23.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.45195 2024-04-01T04:54:24.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:24.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.45195 2024-04-01T04:54:26.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:26.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.45195 2024-04-01T04:54:27.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:27.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.45195 2024-04-01T04:54:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:29.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.45195 2024-04-01T04:54:30.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:30.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.45195 2024-04-01T04:54:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:32.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.45195 2024-04-01T04:54:33.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:33.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.45195 2024-04-01T04:54:35.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:35.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.45195 2024-04-01T04:54:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.45195 2024-04-01T04:54:38.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:38.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.45195 2024-04-01T04:54:39.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:39.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.45195 2024-04-01T04:54:41.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:41.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.45195 2024-04-01T04:54:42.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:42.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.45195 2024-04-01T04:54:44.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:44.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.45195 2024-04-01T04:54:45.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:45.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.45195 2024-04-01T04:54:47.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:47.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.45195 2024-04-01T04:54:48.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:48.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.45195 2024-04-01T04:54:50.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:50.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.45195 2024-04-01T04:54:51.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:51.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.45195 2024-04-01T04:54:52.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:52.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.45195 2024-04-01T04:54:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.45195 2024-04-01T04:54:55.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:55.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.45195 2024-04-01T04:54:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.45195 2024-04-01T04:54:58.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:54:58.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.45195 2024-04-01T04:55:00.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:00.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.45195 2024-04-01T04:55:01.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:01.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.45195 2024-04-01T04:55:03.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:03.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.45195 2024-04-01T04:55:04.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:04.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.45195 2024-04-01T04:55:06.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:06.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.45195 2024-04-01T04:55:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.45195 2024-04-01T04:55:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:09.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.45195 2024-04-01T04:55:10.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:10.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.45195 2024-04-01T04:55:11.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:11.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.45195 2024-04-01T04:55:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:13.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.45195 2024-04-01T04:55:14.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:14.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.45195 2024-04-01T04:55:16.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:16.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.45195 2024-04-01T04:55:18.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:18.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.45195 2024-04-01T04:55:19.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:19.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.45195 2024-04-01T04:55:20.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:20.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.45195 2024-04-01T04:55:22.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:22.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.45195 2024-04-01T04:55:23.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:23.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.45195 2024-04-01T04:55:25.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:25.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.45195 2024-04-01T04:55:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.45195 2024-04-01T04:55:28.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:28.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.45195 2024-04-01T04:55:29.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:29.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.45195 2024-04-01T04:55:31.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:31.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.45195 2024-04-01T04:55:32.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:32.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.45195 2024-04-01T04:55:34.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:34.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.45195 2024-04-01T04:55:35.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:35.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.45195 2024-04-01T04:55:37.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:37.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.45195 2024-04-01T04:55:38.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:38.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.45195 2024-04-01T04:55:39.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:39.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.45195 2024-04-01T04:55:41.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-04-01T04:55:41.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.45195 2024-04-01T04:55:42.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.45195 2024-04-01T04:55:42.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-04-01T04:55:43.014 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 4 2024-04-01T04:55:43.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-04-01T04:55:43.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-04-01T04:55:43.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-04-01T04:55:43.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-04-01T04:55:43.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-04-01T04:55:43.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-04-01T04:55:43.289 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:55:43.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-04-01T04:55:45.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-04-01T04:55:45.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-04-01T04:55:45.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-04-01T04:55:45.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-04-01T04:55:45.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-04-01T04:55:45.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-04-01T04:55:45.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-04-01T04:55:45.588 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:55:45.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-04-01T04:55:47.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-04-01T04:55:47.600 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-04-01T04:55:47.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-04-01T04:55:47.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-04-01T04:55:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-04-01T04:55:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-04-01T04:55:47.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-04-01T04:55:47.881 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:55:47.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-04-01T04:55:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-04-01T04:55:49.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-04-01T04:55:49.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-04-01T04:55:49.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-04-01T04:55:49.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-04-01T04:55:49.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-04-01T04:55:49.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-04-01T04:55:50.199 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:55:50.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-04-01T04:55:52.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-04-01T04:55:52.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-04-01T04:55:52.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-04-01T04:55:52.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-04-01T04:55:52.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-04-01T04:55:52.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-04-01T04:55:52.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-04-01T04:55:52.480 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:55:52.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-04-01T04:55:52.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-04-01T04:55:52.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-04-01T04:55:52.765 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:55:52.765 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-04-01T04:55:45.255742+0000 24'1300 27:1327 [1,0,2,3] 1 [1,0] 1 0'0 2024-04-01T04:53:07.503169+0000 0'0 2024-04-01T04:53:07.503169+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:55:52.765 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:55:52.765 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:55:52.766 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:55:52.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-04-01T04:56:02.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-04-01T04:56:02.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:56:02.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:56:02.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:56:02.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:56:02.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:56:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:56:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-04-01T04:56:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-04-01T04:56:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-04-01T04:56:08.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-04-01T04:56:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-04-01T04:56:08.098 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:46.176+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.098 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:47.150+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.098 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:48.127+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.098 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:49.088+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.098 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:51.077+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.098 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:52.094+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.099 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:55.046+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.099 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:58.082+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.099 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:59.044+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.099 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:06.108+0000 7f520d2bc640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:48.310+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:52.377+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:53.377+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:59.403+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:01.407+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:02.393+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.110 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:04.410+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.111 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:06.436+0000 7f5d8938a640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:54.009+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:57.951+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:58.936+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:59.979+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:01.946+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:03.941+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.113 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:06.992+0000 7f559b66b640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-04-01T04:56:08.115 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:49.155+0000 7f6cdb240640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.116 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:51.139+0000 7f6cdb240640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.116 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:52.137+0000 7f6cdb240640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.116 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:53.120+0000 7f6cdb240640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.116 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:55:58.215+0000 7f6cdb240640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.116 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T04:56:00.158+0000 7f6cdb240640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-04-01T04:56:08.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-04-01T04:56:08.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-04-01T04:56:08.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-04-01T04:56:08.117 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-04-01T04:56:08.117 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-04-01T04:56:08.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-04-01T04:56:08.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-04-01T04:56:08.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-04-01T04:56:08.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-04-01T04:56:08.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-04-01T04:56:08.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-04-01T04:56:08.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-04-01T04:56:08.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-04-01T04:56:08.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-04-01T04:56:08.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-04-01T04:56:08.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-04-01T04:56:08.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-04-01T04:56:08.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:08.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-04-01T04:56:08.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-04-01T04:56:08.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-04-01T04:56:08.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-04-01T04:56:08.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-04-01T04:56:08.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-04-01T04:56:08.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:56:08.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:56:08.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:56:08.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:56:08.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:56:08.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:56:08.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:56:08.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:56:08.151 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:56:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:56:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:56:08.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:56:08.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:56:08.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:56:08.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:56:08.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-04-01T04:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:08.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45195 2024-04-01T04:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:56:08.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-04-01T04:56:08.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-04-01T04:56:08.177 INFO:tasks.workunit.client.0.smithi067.stdout:TEST PASSED 2024-04-01T04:56:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-04-01T04:56:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-04-01T04:56:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-04-01T04:56:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-04-01T04:56:08.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.45195 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:56:08.181 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:56:08.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:56:08.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:56:08.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:56:08.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:56:08.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:56:08.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:56:08.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:56:08.184 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:56:08.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:56:08.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:56:08.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-04-01T04:56:08.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:56:08.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:08.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:08.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45195 2024-04-01T04:56:08.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:56:08.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:56:08.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-04-01T04:56:08.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T04:56:08.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:08.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:08.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.45195 2024-04-01T04:56:08.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T04:56:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T04:56:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T04:56:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-04-01T04:56:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-01T04:56:08.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-04-01T04:56:08.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T04:56:08.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T04:56:08.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T04:56:08.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-04-01T04:56:08.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-01T04:56:08.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T04:56:08.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:08.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:08.575 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:08.576 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:08.576 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:08.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:08.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-01T04:56:08.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T04:56:08.752 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T04:56:08.753 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T04:56:08.753 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:56:08.753 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:08.753 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45195/ceph-mon.a.asok 2024-04-01T04:56:08.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T04:56:08.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45195/ceph-mon.a.asok config get fsid 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45195/ceph-mon.a.asok 2024-04-01T04:56:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T04:56:08.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45195/ceph-mon.a.asok config get mon_host 2024-04-01T04:56:08.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-04-01T04:56:08.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-04-01T04:56:08.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T04:56:08.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T04:56:08.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T04:56:08.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-04-01T04:56:08.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T04:56:09.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T04:56:09.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:09.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:09.719 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:09.719 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:09.720 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:09.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:09.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T04:56:09.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T04:56:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-04-01T04:56:09.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-04-01T04:56:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-04-01T04:56:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-04-01T04:56:09.748 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-04-01T04:56:09.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-04-01T04:56:09.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:09.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:09.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:56:09.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:56:09.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T04:56:09.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:09.754 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:56:09.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97cd2866-17ed-4df3-b7d1-55d5b159b08f 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 97cd2866-17ed-4df3-b7d1-55d5b159b08f' 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 97cd2866-17ed-4df3-b7d1-55d5b159b08f 2024-04-01T04:56:09.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:56:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpPgpmALTgLRAA/yv2ipc70rafCmdTcLjL7w== 2024-04-01T04:56:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpPgpmALTgLRAA/yv2ipc70rafCmdTcLjL7w=="}' 2024-04-01T04:56:09.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97cd2866-17ed-4df3-b7d1-55d5b159b08f -i td/osd-recovery-scrub/0/new.json 2024-04-01T04:56:09.975 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:56:09.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-04-01T04:56:09.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBpPgpmALTgLRAA/yv2ipc70rafCmdTcLjL7w== --osd-uuid 97cd2866-17ed-4df3-b7d1-55d5b159b08f 2024-04-01T04:56:10.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.018+0000 7f548c39a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.026+0000 7f548c39a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.027+0000 7f548c39a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.027+0000 7f548c39a640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-04-01T04:56:10.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.027+0000 7f548c39a640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-04-01T04:56:10.028 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.027+0000 7f548c39a640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-04-01T04:56:10.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:10.118+0000 7f548c39a640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-04-01T04:56:15.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-04-01T04:56:15.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:56:15.623 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T04:56:15.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T04:56:15.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:56:16.009 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T04:56:16.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T04:56:16.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:16.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:56:16.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:56:16.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:56:16.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:16.039+0000 7fc9c8e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:16.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:16.058+0000 7fc9c8e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:16.060 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:16.059+0000 7fc9c8e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:16.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T04:56:16.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:56:16.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T04:56:16.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:56:16.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:56:16.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:16.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:56:16.296 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:56:16.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:16.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:56:16.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:16.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:16.584+0000 7fc9c8e68640 -1 Falling back to public interface 2024-04-01T04:56:17.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:17.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:17.576 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:56:17.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:56:17.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:17.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:56:17.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:18.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:18.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:18.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:56:18.871 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:56:18.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:18.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:56:19.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:19.024+0000 7fc9c8e68640 -1 osd.0 0 log_to_monitors true 2024-04-01T04:56:19.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:20.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:20.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:20.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:56:20.201 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:56:20.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:20.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:56:20.513 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1319584472,v1:127.0.0.1:6803/1319584472] [v2:127.0.0.1:6804/1319584472,v1:127.0.0.1:6805/1319584472] exists,up 97cd2866-17ed-4df3-b7d1-55d5b159b08f 2024-04-01T04:56:20.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:56:20.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:56:20.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:56:20.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:56:20.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:56:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b92fa5e6-57bb-4340-b52b-006030935af5 2024-04-01T04:56:20.522 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 b92fa5e6-57bb-4340-b52b-006030935af5 2024-04-01T04:56:20.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b92fa5e6-57bb-4340-b52b-006030935af5' 2024-04-01T04:56:20.522 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:56:20.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0PgpmlFzMHxAAGLNxAmc2/ZLeloUbNrQDHQ== 2024-04-01T04:56:20.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0PgpmlFzMHxAAGLNxAmc2/ZLeloUbNrQDHQ=="}' 2024-04-01T04:56:20.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b92fa5e6-57bb-4340-b52b-006030935af5 -i td/osd-recovery-scrub/1/new.json 2024-04-01T04:56:20.944 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:56:20.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-04-01T04:56:20.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB0PgpmlFzMHxAAGLNxAmc2/ZLeloUbNrQDHQ== --osd-uuid b92fa5e6-57bb-4340-b52b-006030935af5 2024-04-01T04:56:20.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:20.988+0000 7f93236b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:21.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:21.020+0000 7f93236b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:21.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:21.021+0000 7f93236b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:21.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:21.022+0000 7f93236b5640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-04-01T04:56:21.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:21.022+0000 7f93236b5640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-04-01T04:56:21.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:21.022+0000 7f93236b5640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-04-01T04:56:21.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:21.095+0000 7f93236b5640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-04-01T04:56:26.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-04-01T04:56:26.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:56:26.718 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T04:56:26.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T04:56:26.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:56:27.101 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T04:56:27.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T04:56:27.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:27.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:56:27.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:56:27.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:56:27.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:27.130+0000 7f2e0f411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:27.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:27.132+0000 7f2e0f411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:27.134 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:27.133+0000 7f2e0f411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:56:27.383 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:56:27.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:27.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:56:27.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:27.668 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:27.667+0000 7f2e0f411640 -1 Falling back to public interface 2024-04-01T04:56:28.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:28.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:28.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:56:28.662 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:56:28.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:28.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:56:28.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:29.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:29.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:29.955 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:56:29.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:56:29.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:29.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:56:30.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:30.050+0000 7f2e0f411640 -1 osd.1 0 log_to_monitors true 2024-04-01T04:56:30.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:31.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:31.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:31.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:56:31.248 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:56:31.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:31.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:56:31.567 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/257250609,v1:127.0.0.1:6811/257250609] [v2:127.0.0.1:6812/257250609,v1:127.0.0.1:6813/257250609] exists,up b92fa5e6-57bb-4340-b52b-006030935af5 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:56:31.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:31.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:56:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=12d048a3-a013-484c-8634-0b83f0839a17 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 12d048a3-a013-484c-8634-0b83f0839a17' 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 12d048a3-a013-484c-8634-0b83f0839a17 2024-04-01T04:56:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:56:31.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB/Pgpm9CcVIxAAhaEOzngVBDPsawrs6meVkw== 2024-04-01T04:56:31.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB/Pgpm9CcVIxAAhaEOzngVBDPsawrs6meVkw=="}' 2024-04-01T04:56:31.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 12d048a3-a013-484c-8634-0b83f0839a17 -i td/osd-recovery-scrub/2/new.json 2024-04-01T04:56:31.964 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:56:31.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-04-01T04:56:31.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB/Pgpm9CcVIxAAhaEOzngVBDPsawrs6meVkw== --osd-uuid 12d048a3-a013-484c-8634-0b83f0839a17 2024-04-01T04:56:32.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.006+0000 7f5a43417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:32.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.047+0000 7f5a43417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:32.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.048+0000 7f5a43417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:32.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.048+0000 7f5a43417640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-04-01T04:56:32.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.048+0000 7f5a43417640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-04-01T04:56:32.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.048+0000 7f5a43417640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-04-01T04:56:32.115 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:32.114+0000 7f5a43417640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-04-01T04:56:37.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-04-01T04:56:37.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:56:37.847 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T04:56:37.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T04:56:37.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:56:38.214 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T04:56:38.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T04:56:38.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:38.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:56:38.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:56:38.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:56:38.244 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:38.244+0000 7ffb43cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:38.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:38.246+0000 7ffb43cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:38.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:38.247+0000 7ffb43cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:38.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T04:56:38.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:56:38.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T04:56:38.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:56:38.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:56:38.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:38.500 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:56:38.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:56:38.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:38.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:56:38.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:38.768+0000 7ffb43cd9640 -1 Falling back to public interface 2024-04-01T04:56:38.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:39.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:39.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:39.780 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:56:39.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:56:39.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:39.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:56:40.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:41.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:41.095 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:56:41.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:41.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:56:41.171 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:41.170+0000 7ffb43cd9640 -1 osd.2 0 log_to_monitors true 2024-04-01T04:56:41.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:42.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:42.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:42.387 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:56:42.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:56:42.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:42.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T04:56:42.680 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/410871355,v1:127.0.0.1:6819/410871355] [v2:127.0.0.1:6820/410871355,v1:127.0.0.1:6821/410871355] exists,up 12d048a3-a013-484c-8634-0b83f0839a17 2024-04-01T04:56:42.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:56:42.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:56:42.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:56:42.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:42.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:42.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:56:42.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:42.687 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:56:42.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:56:42.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-04-01T04:56:42.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:56:42.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e4182ea1-e682-4c22-bc83-9953314d6afd 2024-04-01T04:56:42.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e4182ea1-e682-4c22-bc83-9953314d6afd' 2024-04-01T04:56:42.689 INFO:tasks.workunit.client.0.smithi067.stdout:add osd3 e4182ea1-e682-4c22-bc83-9953314d6afd 2024-04-01T04:56:42.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:56:42.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKPgpm3r+/KRAAOdGmTTCmDonKMvUo/7EN4Q== 2024-04-01T04:56:42.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKPgpm3r+/KRAAOdGmTTCmDonKMvUo/7EN4Q=="}' 2024-04-01T04:56:42.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e4182ea1-e682-4c22-bc83-9953314d6afd -i td/osd-recovery-scrub/3/new.json 2024-04-01T04:56:43.033 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:56:43.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-04-01T04:56:43.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCKPgpm3r+/KRAAOdGmTTCmDonKMvUo/7EN4Q== --osd-uuid e4182ea1-e682-4c22-bc83-9953314d6afd 2024-04-01T04:56:43.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.077+0000 7fbe76cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:43.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.118+0000 7fbe76cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.119+0000 7fbe76cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.120+0000 7fbe76cb5640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-04-01T04:56:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.120+0000 7fbe76cb5640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-04-01T04:56:43.120 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.120+0000 7fbe76cb5640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-04-01T04:56:43.191 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:43.191+0000 7fbe76cb5640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-04-01T04:56:48.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-04-01T04:56:48.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:56:48.921 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd3 key to auth repository 2024-04-01T04:56:48.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-01T04:56:48.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:56:49.294 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.3 2024-04-01T04:56:49.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-01T04:56:49.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:49.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:56:49.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:56:49.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:56:49.324 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:49.324+0000 7f5caaaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:49.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:49.326+0000 7f5caaaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:49.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:49.327+0000 7f5caaaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:49.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-01T04:56:49.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:56:49.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T04:56:49.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:56:49.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:56:49.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:49.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:56:49.576 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:56:49.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:49.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:56:49.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:49.844+0000 7f5caaaf9640 -1 Falling back to public interface 2024-04-01T04:56:49.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:50.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:50.861 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:56:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:56:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:56:51.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:52.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:52.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:52.154 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:56:52.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:56:52.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:52.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:56:52.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:52.197+0000 7f5caaaf9640 -1 osd.3 0 log_to_monitors true 2024-04-01T04:56:52.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:56:53.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:56:53.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:56:53.448 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:56:53.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:56:53.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:56:53.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T04:56:53.741 INFO:tasks.workunit.client.0.smithi067.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3812170949,v1:127.0.0.1:6827/3812170949] [v2:127.0.0.1:6828/3812170949,v1:127.0.0.1:6829/3812170949] exists,up e4182ea1-e682-4c22-bc83-9953314d6afd 2024-04-01T04:56:53.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:56:53.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:56:53.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:56:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58bec9e9-465c-4553-b4fd-2c00c09568a3 2024-04-01T04:56:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 58bec9e9-465c-4553-b4fd-2c00c09568a3' 2024-04-01T04:56:53.749 INFO:tasks.workunit.client.0.smithi067.stdout:add osd4 58bec9e9-465c-4553-b4fd-2c00c09568a3 2024-04-01T04:56:53.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:56:53.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVPgpmP7FdLRAARVBCOSY+23gR94Y8MHOHtg== 2024-04-01T04:56:53.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVPgpmP7FdLRAARVBCOSY+23gR94Y8MHOHtg=="}' 2024-04-01T04:56:53.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58bec9e9-465c-4553-b4fd-2c00c09568a3 -i td/osd-recovery-scrub/4/new.json 2024-04-01T04:56:54.211 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T04:56:54.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-04-01T04:56:54.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCVPgpmP7FdLRAARVBCOSY+23gR94Y8MHOHtg== --osd-uuid 58bec9e9-465c-4553-b4fd-2c00c09568a3 2024-04-01T04:56:54.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.253+0000 7f169e59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:54.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.295+0000 7f169e59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:54.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.296+0000 7f169e59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:56:54.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.296+0000 7f169e59e640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-04-01T04:56:54.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.296+0000 7f169e59e640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-04-01T04:56:54.297 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.296+0000 7f169e59e640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-04-01T04:56:54.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:56:54.370+0000 7f169e59e640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-04-01T04:57:00.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-04-01T04:57:00.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:57:00.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-01T04:57:00.085 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd4 key to auth repository 2024-04-01T04:57:00.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:57:00.452 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.4 2024-04-01T04:57:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-01T04:57:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:57:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:57:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:57:00.482 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:00.481+0000 7f6e575a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:00.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:00.483+0000 7f6e575a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:00.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:00.485+0000 7f6e575a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-01T04:57:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:57:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-01T04:57:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:57:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:57:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:00.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:57:00.736 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:57:00.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:00.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T04:57:01.007 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:01.007+0000 7f6e575a8640 -1 Falling back to public interface 2024-04-01T04:57:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:57:02.023 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:57:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T04:57:02.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:57:03.317 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:57:03.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:03.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T04:57:03.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:03.362+0000 7f6e575a8640 -1 osd.4 0 log_to_monitors true 2024-04-01T04:57:03.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:04.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:04.611 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:57:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:57:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:04.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T04:57:04.904 INFO:tasks.workunit.client.0.smithi067.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4144405702,v1:127.0.0.1:6835/4144405702] [v2:127.0.0.1:6836/4144405702,v1:127.0.0.1:6837/4144405702] exists,up 58bec9e9-465c-4553-b4fd-2c00c09568a3 2024-04-01T04:57:04.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:57:04.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:57:04.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:57:04.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:57:04.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:04.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:57:04.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:57:04.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d02900a-e1d7-40b6-8853-885f28e35bfa 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 8d02900a-e1d7-40b6-8853-885f28e35bfa' 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stdout:add osd5 8d02900a-e1d7-40b6-8853-885f28e35bfa 2024-04-01T04:57:04.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:57:04.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgPgpmHigSNxAAzSKvnfvgJ8LFSNDzG4jaOg== 2024-04-01T04:57:04.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgPgpmHigSNxAAzSKvnfvgJ8LFSNDzG4jaOg=="}' 2024-04-01T04:57:04.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d02900a-e1d7-40b6-8853-885f28e35bfa -i td/osd-recovery-scrub/5/new.json 2024-04-01T04:57:05.274 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T04:57:05.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-04-01T04:57:05.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCgPgpmHigSNxAAzSKvnfvgJ8LFSNDzG4jaOg== --osd-uuid 8d02900a-e1d7-40b6-8853-885f28e35bfa 2024-04-01T04:57:05.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.318+0000 7f0276ca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:05.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.350+0000 7f0276ca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.351+0000 7f0276ca2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.351+0000 7f0276ca2640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-04-01T04:57:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.351+0000 7f0276ca2640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-04-01T04:57:05.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.351+0000 7f0276ca2640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-04-01T04:57:05.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:05.427+0000 7f0276ca2640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-04-01T04:57:11.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-04-01T04:57:11.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:57:11.272 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd5 key to auth repository 2024-04-01T04:57:11.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-01T04:57:11.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:57:11.640 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.5 2024-04-01T04:57:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-01T04:57:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:57:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:57:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:57:11.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:11.669+0000 7f126661c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:11.672 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:11.671+0000 7f126661c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:11.673 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:11.672+0000 7f126661c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:57:11.925 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:57:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T04:57:12.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:12.191+0000 7f126661c640 -1 Falling back to public interface 2024-04-01T04:57:12.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:13.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:13.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:13.215 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:57:13.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:57:13.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:13.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T04:57:13.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:14.501+0000 7f126661c640 -1 osd.5 0 log_to_monitors true 2024-04-01T04:57:14.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:14.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:14.507 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:57:14.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:57:14.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:14.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T04:57:14.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:15.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:15.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:15.803 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:57:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:57:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T04:57:16.093 INFO:tasks.workunit.client.0.smithi067.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2940438694,v1:127.0.0.1:6843/2940438694] [v2:127.0.0.1:6844/2940438694,v1:127.0.0.1:6845/2940438694] exists,up 8d02900a-e1d7-40b6-8853-885f28e35bfa 2024-04-01T04:57:16.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:57:16.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:57:16.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:57:16.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:57:16.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:16.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:57:16.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:57:16.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a06150de-8d8c-4418-899c-de6fda6774dd 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 a06150de-8d8c-4418-899c-de6fda6774dd' 2024-04-01T04:57:16.100 INFO:tasks.workunit.client.0.smithi067.stdout:add osd6 a06150de-8d8c-4418-899c-de6fda6774dd 2024-04-01T04:57:16.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:57:16.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsPgpmHVC+BhAAh+xnyJ877QiYAK0d/8krvQ== 2024-04-01T04:57:16.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsPgpmHVC+BhAAh+xnyJ877QiYAK0d/8krvQ=="}' 2024-04-01T04:57:16.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a06150de-8d8c-4418-899c-de6fda6774dd -i td/osd-recovery-scrub/6/new.json 2024-04-01T04:57:16.498 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T04:57:16.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-04-01T04:57:16.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCsPgpmHVC+BhAAh+xnyJ877QiYAK0d/8krvQ== --osd-uuid a06150de-8d8c-4418-899c-de6fda6774dd 2024-04-01T04:57:16.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.541+0000 7fb6d6128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:16.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.584+0000 7fb6d6128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.585+0000 7fb6d6128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.586+0000 7fb6d6128640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-04-01T04:57:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.586+0000 7fb6d6128640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-04-01T04:57:16.587 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.586+0000 7fb6d6128640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-04-01T04:57:16.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:16.650+0000 7fb6d6128640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-04-01T04:57:22.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-04-01T04:57:22.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:57:22.145 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd6 key to auth repository 2024-04-01T04:57:22.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-01T04:57:22.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:57:22.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-01T04:57:22.513 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.6 2024-04-01T04:57:22.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:22.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:57:22.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:57:22.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:57:22.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:22.543+0000 7ff5c0c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:22.545 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:22.545+0000 7ff5c0c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:22.546 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:22.546+0000 7ff5c0c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:22.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-01T04:57:22.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:57:22.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-01T04:57:22.802 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:57:22.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:57:22.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:57:22.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:22.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:57:22.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:22.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T04:57:23.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:23.061+0000 7ff5c0c17640 -1 Falling back to public interface 2024-04-01T04:57:23.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:24.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:24.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:24.117 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:57:24.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:57:24.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:24.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T04:57:24.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:25.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:25.378+0000 7ff5c0c17640 -1 osd.6 0 log_to_monitors true 2024-04-01T04:57:25.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:25.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:25.410 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:57:25.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:57:25.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:25.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T04:57:25.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:26.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:26.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:26.789 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:57:26.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:57:26.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:26.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T04:57:27.111 INFO:tasks.workunit.client.0.smithi067.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2795854463,v1:127.0.0.1:6851/2795854463] [v2:127.0.0.1:6852/2795854463,v1:127.0.0.1:6853/2795854463] exists,up a06150de-8d8c-4418-899c-de6fda6774dd 2024-04-01T04:57:27.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:57:27.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:57:27.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=661a1b45-d54e-4867-be0a-fc37e91c7659 2024-04-01T04:57:27.117 INFO:tasks.workunit.client.0.smithi067.stdout:add osd7 661a1b45-d54e-4867-be0a-fc37e91c7659 2024-04-01T04:57:27.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 661a1b45-d54e-4867-be0a-fc37e91c7659' 2024-04-01T04:57:27.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:57:27.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3PgpmcUetBxAAQrey2Yk3Rk+7XpQKNPcyOQ== 2024-04-01T04:57:27.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3PgpmcUetBxAAQrey2Yk3Rk+7XpQKNPcyOQ=="}' 2024-04-01T04:57:27.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 661a1b45-d54e-4867-be0a-fc37e91c7659 -i td/osd-recovery-scrub/7/new.json 2024-04-01T04:57:27.545 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T04:57:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-04-01T04:57:27.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC3PgpmcUetBxAAQrey2Yk3Rk+7XpQKNPcyOQ== --osd-uuid 661a1b45-d54e-4867-be0a-fc37e91c7659 2024-04-01T04:57:27.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.590+0000 7f8f9f69d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:27.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.630+0000 7f8f9f69d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:27.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.631+0000 7f8f9f69d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:27.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.631+0000 7f8f9f69d640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-04-01T04:57:27.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.631+0000 7f8f9f69d640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-04-01T04:57:27.632 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.631+0000 7f8f9f69d640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-04-01T04:57:27.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:27.700+0000 7f8f9f69d640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-04-01T04:57:33.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-04-01T04:57:33.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:57:33.928 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd7 key to auth repository 2024-04-01T04:57:33.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-04-01T04:57:33.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:57:34.297 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.7 2024-04-01T04:57:34.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-04-01T04:57:34.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=d978f8f6-ab4b-4995-b18a-f293f22f08bf --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45195/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-04-01T04:57:34.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:57:34.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:57:34.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:57:34.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:34.326+0000 7f7e8560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:34.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:34.328+0000 7f7e8560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:34.329+0000 7f7e8560f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:57:34.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-04-01T04:57:34.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:57:34.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-04-01T04:57:34.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:57:34.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:57:34.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:34.580 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:57:34.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:57:34.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:34.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T04:57:34.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:34.853+0000 7f7e8560f640 -1 Falling back to public interface 2024-04-01T04:57:34.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:35.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:35.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:35.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:57:35.889 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:57:35.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:35.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T04:57:36.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:37.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:37.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:37.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:57:37.186 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:57:37.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:37.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T04:57:37.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:57:37.284+0000 7f7e8560f640 -1 osd.7 0 log_to_monitors true 2024-04-01T04:57:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:57:38.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:57:38.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:57:38.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:57:38.503 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:57:38.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:57:38.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T04:57:38.821 INFO:tasks.workunit.client.0.smithi067.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/205346208,v1:127.0.0.1:6859/205346208] [v2:127.0.0.1:6860/205346208,v1:127.0.0.1:6861/205346208] exists,up 661a1b45-d54e-4867-be0a-fc37e91c7659 2024-04-01T04:57:38.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:57:38.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:57:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:57:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-04-01T04:57:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-04-01T04:57:39.199 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'test' created 2024-04-01T04:57:39.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T04:57:40.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-04-01T04:57:40.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T04:57:40.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T04:57:40.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T04:57:40.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T04:57:40.213 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T04:57:40.213 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T04:57:40.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T04:57:40.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T04:57:40.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T04:57:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T04:57:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T04:57:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T04:57:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T04:57:40.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T04:57:40.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:7' 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:40.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T04:57:40.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-04-01T04:57:40.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-04-01T04:57:40.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-04-01T04:57:40.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:40.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T04:57:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-04-01T04:57:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-04-01T04:57:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2024-04-01T04:57:41.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:41.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T04:57:41.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-04-01T04:57:41.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-04-01T04:57:41.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453' 2024-04-01T04:57:41.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:41.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T04:57:41.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-04-01T04:57:41.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-04-01T04:57:41.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931' 2024-04-01T04:57:41.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:41.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T04:57:41.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-04-01T04:57:41.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-04-01T04:57:41.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409' 2024-04-01T04:57:41.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:41.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T04:57:41.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-04-01T04:57:41.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-04-01T04:57:41.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2024-04-01T04:57:41.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:41.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T04:57:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-04-01T04:57:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-04-01T04:57:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2024-04-01T04:57:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T04:57:41.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T04:57:42.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-04-01T04:57:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-04-01T04:57:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2024-04-01T04:57:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:42.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-04-01T04:57:42.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T04:57:42.058 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-04-01T04:57:42.058 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:42.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-04-01T04:57:42.059 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836498 2024-04-01T04:57:42.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-01T04:57:42.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T04:57:42.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-01T04:57:42.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:42.344 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-04-01T04:57:42.344 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:42.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T04:57:42.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-04-01T04:57:42.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-04-01T04:57:42.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-01T04:57:42.347 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672975 2024-04-01T04:57:42.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T04:57:42.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-04-01T04:57:42.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:42.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-04-01T04:57:42.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:42.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T04:57:42.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-04-01T04:57:42.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:42.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-04-01T04:57:42.635 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509453 2024-04-01T04:57:42.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-01T04:57:42.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T04:57:42.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-04-01T04:57:42.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:42.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-04-01T04:57:42.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:42.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T04:57:42.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-04-01T04:57:42.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:42.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-04-01T04:57:42.930 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345931 2024-04-01T04:57:42.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-04-01T04:57:42.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T04:57:43.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-04-01T04:57:43.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:43.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-04-01T04:57:43.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:43.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T04:57:43.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-04-01T04:57:43.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:43.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-04-01T04:57:43.217 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.4 seq 107374182409 2024-04-01T04:57:43.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-04-01T04:57:43.217 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T04:57:43.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-04-01T04:57:43.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:43.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-04-01T04:57:43.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:43.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T04:57:43.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-04-01T04:57:43.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:43.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-04-01T04:57:43.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-04-01T04:57:43.502 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.5 seq 128849018887 2024-04-01T04:57:43.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T04:57:43.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-04-01T04:57:43.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:43.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2024-04-01T04:57:43.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:43.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T04:57:43.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2024-04-01T04:57:43.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:43.788 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.6 seq 150323855365 2024-04-01T04:57:43.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-04-01T04:57:43.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-04-01T04:57:43.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T04:57:44.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-04-01T04:57:44.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T04:57:44.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-04-01T04:57:44.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T04:57:44.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T04:57:44.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-04-01T04:57:44.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T04:57:44.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-04-01T04:57:44.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-04-01T04:57:44.139 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.7 seq 171798691842 2024-04-01T04:57:44.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T04:57:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-04-01T04:57:44.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T04:57:44.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T04:57:44.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T04:57:44.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T04:57:45.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-04-01T04:57:45.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T04:57:45.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T04:57:45.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T04:57:45.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-04-01T04:57:45.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T04:57:45.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T04:57:45.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-04-01T04:57:45.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-04-01T04:57:45.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-04-01T04:57:45.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-04-01T04:57:45.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.45195 bs=1M count=50 2024-04-01T04:57:45.881 INFO:tasks.workunit.client.0.smithi067.stderr:50+0 records in 2024-04-01T04:57:45.881 INFO:tasks.workunit.client.0.smithi067.stderr:50+0 records out 2024-04-01T04:57:45.881 INFO:tasks.workunit.client.0.smithi067.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.138029 s, 380 MB/s 2024-04-01T04:57:45.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-04-01T04:57:45.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:45.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.45195 2024-04-01T04:57:47.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:47.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.45195 2024-04-01T04:57:48.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:48.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.45195 2024-04-01T04:57:50.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:50.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.45195 2024-04-01T04:57:51.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:51.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.45195 2024-04-01T04:57:53.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:53.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.45195 2024-04-01T04:57:54.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:54.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.45195 2024-04-01T04:57:55.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:55.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.45195 2024-04-01T04:57:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.45195 2024-04-01T04:57:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:57:58.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.45195 2024-04-01T04:58:00.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:00.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.45195 2024-04-01T04:58:01.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:01.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.45195 2024-04-01T04:58:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:03.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.45195 2024-04-01T04:58:04.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:04.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.45195 2024-04-01T04:58:05.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:05.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.45195 2024-04-01T04:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:07.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.45195 2024-04-01T04:58:08.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:08.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.45195 2024-04-01T04:58:10.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:10.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.45195 2024-04-01T04:58:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.45195 2024-04-01T04:58:13.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:13.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.45195 2024-04-01T04:58:14.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:14.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.45195 2024-04-01T04:58:15.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:15.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.45195 2024-04-01T04:58:17.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:17.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.45195 2024-04-01T04:58:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.45195 2024-04-01T04:58:20.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:20.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.45195 2024-04-01T04:58:22.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:22.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.45195 2024-04-01T04:58:23.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:23.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.45195 2024-04-01T04:58:25.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:25.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.45195 2024-04-01T04:58:26.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:26.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.45195 2024-04-01T04:58:27.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:27.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.45195 2024-04-01T04:58:29.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:29.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.45195 2024-04-01T04:58:30.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:30.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.45195 2024-04-01T04:58:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.45195 2024-04-01T04:58:33.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:34.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.45195 2024-04-01T04:58:35.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:35.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.45195 2024-04-01T04:58:36.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:36.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.45195 2024-04-01T04:58:38.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:38.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.45195 2024-04-01T04:58:39.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:39.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.45195 2024-04-01T04:58:40.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:40.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.45195 2024-04-01T04:58:42.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-04-01T04:58:42.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.45195 2024-04-01T04:58:43.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.45195 2024-04-01T04:58:43.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-04-01T04:58:44.186 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 3 2024-04-01T04:58:44.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-04-01T04:57:40.989437+0000 48'26 48:55 [5] 5 [5] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.989624+0000 48'13 48:41 [3] 3 [3] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:41.326156+0000 48'13 48:27 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:41.325964+0000 48'13 48:30 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:41.232737+0000 48'13 48:30 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:57:41.325311+0000 0'0 48:17 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.473 INFO:tasks.workunit.client.0.smithi067.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-04-01T04:57:40.890902+0000 48'26 48:47 [5] 5 [5] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.577278+0000 48'13 48:34 [6] 6 [6] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.6 1 0 0 0 0 37748736 0 0 9 0 9 active+clean 2024-04-01T04:57:40.392771+0000 48'9 48:29 [6] 6 [6] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-04-01T04:57:40.518562+0000 48'39 48:60 [4] 4 [4] 4 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-04-01T04:57:40.989169+0000 48'52 48:74 [3] 3 [3] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-04-01T04:57:40.392742+0000 48'52 48:73 [4] 4 [4] 4 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.006589+0000 48'13 48:38 [0] 0 [0] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:57:40.890878+0000 0'0 48:20 [3] 3 [3] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.890887+0000 48'13 48:42 [5] 5 [5] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.006710+0000 48'13 48:31 [0] 0 [0] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.392808+0000 48'13 48:39 [7] 7 [7] 7 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.518526+0000 48'13 48:30 [7] 7 [7] 7 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.474 INFO:tasks.workunit.client.0.smithi067.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:41.326008+0000 48'13 48:29 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.577186+0000 48'13 48:31 [6] 6 [6] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.989341+0000 48'13 48:33 [5] 5 [5] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.006694+0000 48'13 48:33 [0] 0 [0] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.12 2 0 0 0 0 85983232 0 0 21 0 21 active+clean 2024-04-01T04:57:39.909314+0000 48'21 48:40 [0] 0 [0] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.518528+0000 48'13 48:30 [4] 4 [4] 4 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:57:40.291874+0000 0'0 48:18 [2] 2 [2] 2 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-04-01T04:57:41.326083+0000 48'39 48:54 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:57:40.577386+0000 0'0 48:17 [6] 6 [6] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.006536+0000 48'13 48:32 [0] 0 [0] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:57:40.890895+0000 0'0 48:19 [5] 5 [5] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.989273+0000 48'13 48:31 [3] 3 [3] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.475 INFO:tasks.workunit.client.0.smithi067.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:57:41.326219+0000 0'0 48:15 [1] 1 [1] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.476 INFO:tasks.workunit.client.0.smithi067.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-04-01T04:57:40.006806+0000 48'13 48:32 [0] 0 [0] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:44.476 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:58:44.476 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:58:44.476 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:58:44.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-04-01T04:58:44.623 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:58:44.623 INFO:tasks.workunit.client.0.smithi067.stdout: "osd_recovery_max_active": "0" 2024-04-01T04:58:44.624 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:58:44.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-04-01T04:58:44.884 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:58:44.885 INFO:tasks.workunit.client.0.smithi067.stdout: "osd_recovery_max_active_hdd": "3" 2024-04-01T04:58:44.885 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:58:44.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-04-01T04:58:45.211 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T04:58:45.211 INFO:tasks.workunit.client.0.smithi067.stdout: "osd_recovery_max_active_ssd": "10" 2024-04-01T04:58:45.211 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T04:58:45.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-04-01T04:58:45.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-04-01T04:58:45.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-04-01T04:58:45.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-04-01T04:58:45.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-04-01T04:58:45.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-04-01T04:58:45.659 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:58:45.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-04-01T04:58:45.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-04-01T04:58:47.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-04-01T04:58:47.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-04-01T04:58:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-04-01T04:58:47.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-04-01T04:58:47.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-04-01T04:58:47.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-04-01T04:58:47.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-04-01T04:58:47.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-04-01T04:58:47.956 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:58:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-04-01T04:58:47.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-04-01T04:58:49.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-04-01T04:58:49.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-04-01T04:58:49.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-04-01T04:58:49.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-04-01T04:58:49.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-04-01T04:58:49.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-04-01T04:58:49.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-04-01T04:58:49.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-04-01T04:58:50.250 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:58:50.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 3 -ge 2 2024-04-01T04:58:50.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-04-01T04:58:50.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 activating+degraded 2024-04-01T04:58:45.933157+0000 48'26 51:60 [5,0,2] 5 [5,0,2] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.624522+0000 48'13 51:51 [3,0,7] 3 [3,0,7] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 activating+degraded 2024-04-01T04:58:46.329187+0000 48'26 51:45 [1,3,0] 1 [1,3,0] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-04-01T04:58:46.248093+0000 48'13 51:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.519256+0000 48'13 51:39 [1,6,2] 1 [1,6,2] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.528 INFO:tasks.workunit.client.0.smithi067.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:58:46.155540+0000 0'0 51:27 [1,4,5] 1 [1,4,5] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 activating 2024-04-01T04:58:45.612641+0000 48'26 51:51 [5,4,3] 5 [5,4,3] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.155357+0000 48'13 51:43 [6,7,3] 6 [6,7,3] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.330011+0000 48'13 51:43 [6,7,3] 6 [6,7,3] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-04-01T04:58:46.288320+0000 48'39 51:70 [4,2,6] 4 [4,2,6] 4 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovering+degraded 2024-04-01T04:58:46.520329+0000 48'52 51:85 [3,0,6] 3 [3,0,6] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovering+degraded 2024-04-01T04:58:46.155842+0000 48'52 51:84 [4,1,2] 4 [4,1,2] 4 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-04-01T04:58:45.369688+0000 48'13 50:42 [0,7,1] 0 [0,7,1] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:58:46.205473+0000 0'0 51:31 [3,0,4] 3 [3,0,4] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:50.529 INFO:tasks.workunit.client.0.smithi067.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-04-01T04:58:45.933212+0000 48'13 51:47 [5,0,4] 5 [5,0,4] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-04-01T04:58:45.222903+0000 48'13 50:40 [0,3,4] 0 [0,3,4] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.792156+0000 48'13 51:49 [7,0,2] 7 [7,0,2] 7 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.330039+0000 48'13 51:39 [7,4,2] 7 [7,4,2] 7 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.155158+0000 48'13 51:38 [1,4,2] 1 [1,4,2] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-04-01T04:58:46.556090+0000 48'13 51:36 [6,0,3] 6 [6,0,3] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-04-01T04:58:45.614592+0000 48'13 51:37 [5,7,6] 5 [5,7,6] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-04-01T04:58:44.918515+0000 48'13 50:36 [0,1,3] 0 [0,1,3] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 activating+degraded 2024-04-01T04:58:45.220742+0000 48'26 50:49 [0,1,6] 0 [0,1,6] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.287995+0000 48'13 51:39 [4,6,3] 4 [4,6,3] 4 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:58:46.623523+0000 0'0 51:29 [2,0,4] 2 [2,0,4] 2 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-04-01T04:58:46.154121+0000 48'39 51:63 [1,5,3] 1 [1,5,3] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:58:46.205471+0000 0'0 51:28 [6,0,7] 6 [6,0,7] 6 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 periodic scrub scheduled @ 2024-04-02T04:57:39.147477+0000 0 0 2024-04-01T04:58:50.530 INFO:tasks.workunit.client.0.smithi067.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-04-01T04:58:44.918461+0000 48'13 50:35 [0,4,5] 0 [0,4,5] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 activating 2024-04-01T04:58:45.612302+0000 0'0 51:24 [5,2,3] 5 [5,2,3] 5 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-04-01T04:58:46.624663+0000 48'13 51:40 [3,0,4] 3 [3,0,4] 3 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-01T04:58:46.154700+0000 0'0 51:25 [1,7,4] 1 [1,7,4] 1 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-04-01T04:58:45.220554+0000 48'13 50:36 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-01T04:57:39.147477+0000 0'0 2024-04-01T04:57:39.147477+0000 0 0 no scrub is scheduled 0 0 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T04:58:50.531 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T04:58:50.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-04-01T04:58:50.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-04-01T04:58:50.539 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-04-01T04:58:50.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-04-01T04:58:50.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-04-01T04:58:50.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-04-01T04:58:50.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66680"' 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66680' 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66683: /' 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66685"' 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66685' 2024-04-01T04:58:50.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66690: /' 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66693"' 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66693' 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-04-01T04:58:50.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66697: /' 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66699"' 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66699' 2024-04-01T04:58:50.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66703: /' 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66707"' 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66707' 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66711: /' 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66712"' 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66712' 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66717: /' 2024-04-01T04:58:50.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66719"' 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66719' 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66726"' 2024-04-01T04:58:50.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66726' 2024-04-01T04:58:50.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66728"' 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66728' 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-04-01T04:58:50.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66731: /' 2024-04-01T04:58:50.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-04-01T04:58:50.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-04-01T04:58:50.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-04-01T04:58:50.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-04-01T04:58:50.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66722: /' 2024-04-01T04:58:50.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66743"' 2024-04-01T04:58:50.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66743' 2024-04-01T04:58:50.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66733: /' 2024-04-01T04:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-04-01T04:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-04-01T04:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66747"' 2024-04-01T04:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66747' 2024-04-01T04:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66746: /' 2024-04-01T04:58:50.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-04-01T04:58:50.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-04-01T04:58:50.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-04-01T04:58:50.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66751"' 2024-04-01T04:58:50.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66751' 2024-04-01T04:58:50.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-04-01T04:58:50.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66750: /' 2024-04-01T04:58:50.623 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-04-01T04:58:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-04-01T04:58:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-04-01T04:58:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66758: /' 2024-04-01T04:58:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66760"' 2024-04-01T04:58:50.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66760' 2024-04-01T04:58:50.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-04-01T04:58:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-04-01T04:58:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66764"' 2024-04-01T04:58:50.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66764' 2024-04-01T04:58:50.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-04-01T04:58:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-04-01T04:58:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66766"' 2024-04-01T04:58:50.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66766' 2024-04-01T04:58:50.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-04-01T04:58:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-04-01T04:58:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-04-01T04:58:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66773: /' 2024-04-01T04:58:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66775"' 2024-04-01T04:58:50.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66775' 2024-04-01T04:58:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66778: /' 2024-04-01T04:58:50.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-04-01T04:58:50.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66769: /' 2024-04-01T04:58:50.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-04-01T04:58:50.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-04-01T04:58:50.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-04-01T04:58:50.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66781"' 2024-04-01T04:58:50.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66781' 2024-04-01T04:58:50.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-04-01T04:58:50.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66784: /' 2024-04-01T04:58:50.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-04-01T04:58:50.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66789"' 2024-04-01T04:58:50.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66789' 2024-04-01T04:58:50.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-04-01T04:58:50.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-04-01T04:58:50.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66797"' 2024-04-01T04:58:50.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66797' 2024-04-01T04:58:50.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-04-01T04:58:50.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-04-01T04:58:50.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66794: /' 2024-04-01T04:58:50.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66803: /' 2024-04-01T04:58:50.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-04-01T04:58:50.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-04-01T04:58:50.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-04-01T04:58:50.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66809"' 2024-04-01T04:58:50.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66809' 2024-04-01T04:58:50.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-04-01T04:58:50.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-04-01T04:58:50.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66811"' 2024-04-01T04:58:50.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66811' 2024-04-01T04:58:50.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-04-01T04:58:50.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-04-01T04:58:50.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66814"' 2024-04-01T04:58:50.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66814' 2024-04-01T04:58:50.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66817"' 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66817' 2024-04-01T04:58:50.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-04-01T04:58:50.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-04-01T04:58:50.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-04-01T04:58:50.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66823"' 2024-04-01T04:58:50.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66823' 2024-04-01T04:58:50.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-04-01T04:58:50.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-04-01T04:58:50.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-04-01T04:58:50.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66832"' 2024-04-01T04:58:50.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66832' 2024-04-01T04:58:50.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-04-01T04:58:50.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66820: /' 2024-04-01T04:58:50.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-04-01T04:58:50.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-04-01T04:58:50.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66831: /' 2024-04-01T04:58:50.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66824: /' 2024-04-01T04:58:50.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66835: /' 2024-04-01T04:58:50.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66829: /' 2024-04-01T04:58:50.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66863: /' 2024-04-01T04:58:50.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-04-01T04:58:50.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-04-01T04:58:50.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-04-01T04:58:50.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66866"' 2024-04-01T04:58:50.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66866' 2024-04-01T04:58:50.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-04-01T04:58:50.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66855: /' 2024-04-01T04:58:50.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-04-01T04:58:50.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-04-01T04:58:50.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66882"' 2024-04-01T04:58:50.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66882' 2024-04-01T04:58:50.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-04-01T04:58:50.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-04-01T04:58:50.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-04-01T04:58:50.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66908"' 2024-04-01T04:58:50.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66908' 2024-04-01T04:58:50.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66885: /' 2024-04-01T04:58:50.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66892: /' 2024-04-01T04:58:50.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-04-01T04:58:50.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-04-01T04:58:50.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-04-01T04:58:50.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66931"' 2024-04-01T04:58:50.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66931' 2024-04-01T04:58:50.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66927: /' 2024-04-01T04:58:50.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-04-01T04:58:50.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-04-01T04:58:50.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-04-01T04:58:50.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66956"' 2024-04-01T04:58:50.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66956' 2024-04-01T04:58:50.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-04-01T04:58:50.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66955: /' 2024-04-01T04:58:50.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-04-01T04:58:50.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-04-01T04:58:50.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66972"' 2024-04-01T04:58:50.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66972' 2024-04-01T04:58:50.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-04-01T04:58:50.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-04-01T04:58:50.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66984: /' 2024-04-01T04:58:50.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-04-01T04:58:50.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66999"' 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66999' 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 66680 66685 66693 66699 66707 66712 66719 66726 66728 66743 66747 66751 66760 66764 66766 66775 66781 66789 66797 66809 66811 66814 66817 66823 66832 66866 66882 66908 66931 66956 66972 66999' 2024-04-01T04:58:50.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-04-01T04:58:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:58:50.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66680 2024-04-01T04:58:50.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/67001: /' 2024-04-01T04:58:50.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-04-01T04:58:50.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-04-01T04:58:50.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/67040: /' 2024-04-01T04:58:56.668 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-04-01T04:58:56.668 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-04-01T04:58:56.668 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-04-01T04:58:56.668 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:56.668 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-04-01T04:58:56.668 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: instructing pg 1.4 on osd.3 to scrub 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: dumped pgs 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:56.669 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:56.670 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:56.860 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: instructing pg 1.3 on osd.4 to scrub 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: dumped pgs 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:56.861 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T04:58:56.862 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T04:58:56.862 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:56.862 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66703: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:57.863 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: instructing pg 1.5 on osd.4 to scrub 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: dumped pgs 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:57.864 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:66717: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:57.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:58.215 INFO:tasks.workunit.client.0.smithi067.stderr:_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.216 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:58.216 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:58.216 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-04-01T04:58:58.216 INFO:tasks.workunit.client.0.smithi067.stderr:66711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-04-01T04:58:58.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-04-01T04:58:58.390 INFO:tasks.workunit.client.0.smithi067.stderr:lect(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T04:58:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-04-01T04:58:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:66703: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-04-01T04:58:58.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-04-01T04:58:58.569 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-04-01T04:58:58.569 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:58.570 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: instructing pg 1.a on osd.1 to scrub 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: dumped pgs 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:58.571 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:66750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-04-01T04:58:58.671 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: instructing pg 1.1c on osd.0 to scrub 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: dumped pgs 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.672 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-04-01T04:58:58.726 INFO:tasks.workunit.client.0.smithi067.stderr:66955: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: g66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-04-01T04:58:58.726 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-04-01T04:58:58.726 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-04-01T04:58:58.726 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: instructing pg 1.16 on osd.6 to scrub 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-04-01T04:58:58.727 INFO:tasks.workunit.client.0.smithi067.stderr:66829: dumped pgs 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-04-01T04:58:58.728 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:58:55.019127+0000 '>' '' 2024-04-01T04:58:58.729 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:58.729 INFO:tasks.workunit.client.0.smithi067.stderr:66829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:58.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.770 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: instructing pg 1.1 on osd.5 to scrub 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: dumped pgs 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:58.771 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:66690: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:58.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:58.908 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-04-01T04:58:58.908 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-04-01T04:58:58.909 INFO:tasks.workunit.client.0.smithi067.stderr:66733: instructing pg 1.8 on osd.5 to scrub 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: dumped pgs 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:58.910 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:66733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:58.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.174 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-04-01T04:58:59.174 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: instructing pg 1.7 on osd.6 to scrub 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.175 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: dumped pgs 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-04-01T04:58:59.176 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.177 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.177 INFO:tasks.workunit.client.0.smithi067.stderr:66731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.184 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: instructing pg 1.2 on osd.0 to scrub 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: dumped pgs 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.185 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:66697: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.223 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: instructing pg 1.b on osd.1 to scrub 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: dumped pgs 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.224 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:66758: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: instructing pg 1.14 on osd.2 to scrub 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: dumped pgs 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-04-01T04:58:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:66855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:58:56.358268+0000 '>' '' 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:66855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.477 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-04-01T04:58:59.477 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: instructing pg 1.17 on osd.0 to scrub 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.478 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: dumped pgs 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-04-01T04:58:59.479 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.480 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.480 INFO:tasks.workunit.client.0.smithi067.stderr:66863: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.536 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.537 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: instructing pg 1.18 on osd.5 to scrub 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: dumped pgs 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.538 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:58:55.634109+0000 '>' '' 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:66835: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.540 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: instructing pg 1.15 on osd.1 to scrub 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: dumped pgs 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.541 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:66820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.576 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: instructing pg 1.1e on osd.3 to scrub 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: dumped pgs 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:67001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.641 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: instructing pg 1.13 on osd.4 to scrub 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: dumped pgs 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-04-01T04:58:59.642 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:66831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.669 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-04-01T04:58:59.670 INFO:tasks.workunit.client.0.smithi067.stderr:66892: instructing pg 1.1a on osd.1 to scrub 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: dumped pgs 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.671 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66892: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-04-01T04:58:59.672 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: instructing pg 1.12 on osd.0 to scrub 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: dumped pgs 2024-04-01T04:58:59.673 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.674 INFO:tasks.workunit.client.0.smithi067.stderr:-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:66824: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-04-01T04:58:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-04-01T04:58:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-04-01T04:58:59.677 INFO:tasks.workunit.client.0.smithi067.stderr:66778: instructing pg 1.e on osd.1 to scrub 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: dumped pgs 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.678 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:66778: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:58:59.685 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-04-01T04:58:59.685 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: instructing pg 1.f on osd.6 to scrub 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:58:59.686 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: dumped pgs 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:58:59.687 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-04-01T04:58:59.688 INFO:tasks.workunit.client.0.smithi067.stderr:66784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:58:59.688 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:58:59.688 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:58:59.688 INFO:tasks.workunit.client.0.smithi067.stderr:66784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:58:59.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.168 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-04-01T04:59:00.168 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-04-01T04:59:00.168 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: instructing pg 1.9 on osd.1 to scrub 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: dumped pgs 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.170 INFO:tasks.workunit.client.0.smithi067.stderr:66746: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-04-01T04:59:00.171 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.171 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.171 INFO:tasks.workunit.client.0.smithi067.stderr:66746: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.236 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: instructing pg 1.c on osd.3 to scrub 2024-04-01T04:59:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: dumped pgs 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-04-01T04:59:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.239 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.239 INFO:tasks.workunit.client.0.smithi067.stderr:66773: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-04-01T04:59:00.239 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:58:57.052336+0000 '>' '' 2024-04-01T04:59:00.239 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.239 INFO:tasks.workunit.client.0.smithi067.stderr:66773: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.286 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-04-01T04:59:00.286 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-04-01T04:59:00.286 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.287 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: instructing pg 1.d on osd.7 to scrub 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: dumped pgs 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.288 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:66769: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.315 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-04-01T04:59:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:66955: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-04-01T04:59:00.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-04-01T04:59:00.349 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-04-01T04:59:00.349 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-04-01T04:59:00.349 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-04-01T04:59:00.349 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: instructing pg 1.10 on osd.5 to scrub 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-04-01T04:59:00.350 INFO:tasks.workunit.client.0.smithi067.stderr:66794: dumped pgs 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.351 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.352 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.352 INFO:tasks.workunit.client.0.smithi067.stderr:66794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-04-01T04:59:00.361 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: instructing pg 1.0 on osd.7 to scrub 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: dumped pgs 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.362 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66685 2024-04-01T04:59:00.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.366 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-04-01T04:59:00.366 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: instructing pg 1.1f on osd.5 to scrub 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: dumped pgs 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:67040: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66693 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:67040: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66699 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-04-01T04:59:00.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66707 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66712 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66719 2024-04-01T04:59:00.401 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.402 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: instructing pg 1.1b on osd.0 to scrub 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: dumped pgs 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-04-01T04:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: instructing pg 1.1d on osd.1 to scrub 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: dumped pgs 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.405 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:66984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-04-01T04:59:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: instructing pg 1.11 on osd.0 to scrub 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: dumped pgs 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-04-01T04:59:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:66803: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.439 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-04-01T04:59:00.439 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-04-01T04:59:00.439 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-04-01T04:59:00.439 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.439 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: instructing pg 1.6 on osd.6 to scrub 2024-04-01T04:59:00.440 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: dumped pgs 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66726 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66728 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66743 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66747 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66751 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66760 2024-04-01T04:59:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66764 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66766 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66775 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66781 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66789 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66797 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66809 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66811 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66814 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66817 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66823 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66832 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66866 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-04-01T04:59:00.449 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: instructing pg 1.19 on osd.3 to scrub 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-04-01T04:59:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: dumped pgs 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-04-01T04:57:39.147477+0000 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-04-01T04:59:00.451 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-04-01T04:57:39.147477+0000 '>' '' 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:66885: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66882 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66908 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66931 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 2 + 1 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=3 2024-04-01T04:59:00.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66956 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66972 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66999 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-04-01T04:59:00.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 3 -eq 0 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.5.pid 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.6.pid 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.1.pid 2024-04-01T04:59:00.455 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.0.pid 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.4.pid 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.3.pid 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.7.pid 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:td/osd-recovery-scrub/osd.2.pid' 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53855 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:61765 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:63141 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:56261 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:53939 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:54885 2024-04-01T04:59:00.456 INFO:tasks.workunit.client.0.smithi067.stderr:60389 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:59013 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:64517 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:57637' 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53855 61765 63141 56261 53939 54885 60389 59013 64517 57637 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:59:00.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:59:01.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:59:01.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-04-01T04:59:01.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-04-01T04:59:01.751 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-04-01T04:59:01.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-04-01T04:59:01.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-04-01T04:59:01.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-04-01T04:59:01.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-04-01T04:59:01.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-04-01T04:59:01.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-04-01T04:59:01.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-04-01T04:59:01.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-04-01T04:59:01.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-04-01T04:59:01.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-04-01T04:59:01.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-04-01T04:59:01.783 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-04-01T04:59:01.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-04-01T04:59:01.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-04-01T04:59:01.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-04-01T04:59:01.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-04-01T04:59:01.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-04-01T04:59:01.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-04-01T04:59:01.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-04-01T04:59:01.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-04-01T04:59:01.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T04:59:01.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-04-01T04:59:01.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:59:01.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:59:01.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:59:01.818 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:59:01.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:59:01.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:59:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:59:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:59:01.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:59:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:59:01.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:59:01.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:01.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:59:01.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:59:01.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-04-01T04:59:01.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:59:01.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:01.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:59:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45195 2024-04-01T04:59:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:59:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:59:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-04-01T04:59:01.850 INFO:tasks.workunit.client.0.smithi067.stdout:TEST PASSED 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:59:01.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:59:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:59:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:59:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:59:01.852 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:59:01.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:59:01.853 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:59:01.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:59:01.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:59:01.854 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:59:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:59:01.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:59:01.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:01.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:59:01.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-01T04:59:01.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-04-01T04:59:01.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:59:01.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:01.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45195 2024-04-01T04:59:01.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45195 2024-04-01T04:59:01.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:59:01.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:59:01.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-04-01T04:59:01.861 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-01T04:59:01.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-01T04:59:01.929 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-04-01T04:59:01.929 DEBUG:teuthology.orchestra.run.smithi067:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-04-01T04:59:01.990 INFO:tasks.workunit.client.0.smithi067.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-01T04:59:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-01T04:59:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-04-01T04:59:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-04-01T04:59:01.997 INFO:tasks.workunit.client.0.smithi067.stdout:This test is disabled 2024-04-01T04:59:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-04-01T04:59:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-04-01T04:59:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-04-01T04:59:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-04-01T04:59:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-01T04:59:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:59:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:59:02.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:59:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:59:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:59:02.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-01T04:59:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70200 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70200 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:59:02.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-04-01T04:59:02.005 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-01T04:59:02.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-01T04:59:02.071 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-04-01T04:59:02.071 DEBUG:teuthology.orchestra.run.smithi067:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ TIMEOUT=300 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ MAX_TIMEOUT=15 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ PG_NUM=4 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ TMPDIR=/tmp 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-04-01T04:59:02.131 INFO:tasks.workunit.client.0.smithi067.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:++ type xmlstarlet 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:++ XMLSTARLET=xmlstarlet 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:+++ uname 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' Linux = FreeBSD ']' 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:++ SED=sed 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:++ AWK=awk 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:+++ stty -a 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:+++ head -1 2024-04-01T04:59:02.132 INFO:tasks.workunit.client.0.smithi067.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-04-01T04:59:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-01T04:59:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:++ termwidth= 2024-04-01T04:59:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-04-01T04:59:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:++ DIFFCOLOPTS='-y ' 2024-04-01T04:59:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:++ KERNCORE=kernel.core_pattern 2024-04-01T04:59:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:++ EXTRA_OPTS= 2024-04-01T04:59:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:++ test '' = TESTS 2024-04-01T04:59:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-04-01T04:59:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ uname 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' Linux = FreeBSD ']' 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ use_ec_overwrite=true 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ getjson=no 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+ jqfilter='def walk(f): 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T04:59:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T04:59:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T04:59:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T04:59:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T04:59:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T04:59:02.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ main osd-scrub-repair 2024-04-01T04:59:02.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ local dir=td/osd-scrub-repair 2024-04-01T04:59:02.144 INFO:tasks.workunit.client.0.smithi067.stderr:+ shift 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:+ shopt -s -o xtrace 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-04-01T04:59:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none ' 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-04-01T04:59:02.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-04-01T04:59:02.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_bluestore_basic 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_bluestore_failed 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_bluestore_scrub 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_bluestore_tag 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_erasure_coded_appends 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_and_repair_lrc_appends 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-04-01T04:59:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_and_repair_replicated 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_scrub_erasure_appends 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_scrub_replicated 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_corrupt_snapset_scrub_rep 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_list_missing_erasure_coded_appends 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_list_missing_erasure_coded_overwrites 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_periodic_scrub_replicated 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_repair_stats 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_repair_stats_ec 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_request_scrub_priority 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_scrub_warning 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_skip_non_repair_during_recovery 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_unfound_erasure_coded_appends 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:TEST_unfound_erasure_coded_overwrites' 2024-04-01T04:59:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T04:59:02.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T04:59:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T04:59:02.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T04:59:02.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T04:59:02.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T04:59:02.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:02.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:02.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-01T04:59:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T04:59:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-04-01T04:59:02.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T04:59:02.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:59:02.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:59:02.481 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:59:02.482 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:02.482 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:02.482 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T04:59:02.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-04-01T04:59:02.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T04:59:02.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T04:59:02.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T04:59:02.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T04:59:02.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T04:59:02.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T04:59:02.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T04:59:03.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T04:59:03.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T04:59:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:59:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:59:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:59:03.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-04-01T04:59:03.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T04:59:03.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:59:03.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4e3a9535-7904-40b3-aecd-ec0037c569de 2024-04-01T04:59:03.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4e3a9535-7904-40b3-aecd-ec0037c569de' 2024-04-01T04:59:03.674 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 4e3a9535-7904-40b3-aecd-ec0037c569de 2024-04-01T04:59:03.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:59:03.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXPwpmqEjxKBAA/I7bbWvUXPBBQeSnaHZvsg== 2024-04-01T04:59:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXPwpmqEjxKBAA/I7bbWvUXPBBQeSnaHZvsg=="}' 2024-04-01T04:59:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4e3a9535-7904-40b3-aecd-ec0037c569de -i td/osd-scrub-repair/0/new.json 2024-04-01T04:59:03.895 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:59:03.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T04:59:03.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAXPwpmqEjxKBAA/I7bbWvUXPBBQeSnaHZvsg== --osd-uuid 4e3a9535-7904-40b3-aecd-ec0037c569de 2024-04-01T04:59:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:03.935+0000 7fe07560c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:03.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:03.945+0000 7fe07560c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:03.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:03.947+0000 7fe07560c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:03.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:03.947+0000 7fe07560c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T04:59:03.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:03.947+0000 7fe07560c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T04:59:03.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:03.947+0000 7fe07560c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T04:59:04.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:04.037+0000 7fe07560c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T04:59:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T04:59:09.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:59:09.550 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T04:59:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T04:59:09.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:59:09.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T04:59:09.947 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T04:59:09.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-04-01T04:59:09.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:59:09.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:59:09.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:59:09.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:09.974+0000 7f3c1c80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:10.004 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:10.003+0000 7f3c1c80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:10.005 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:10.004+0000 7f3c1c80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:10.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T04:59:10.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:59:10.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T04:59:10.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:59:10.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:59:10.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:10.234 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:59:10.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:59:10.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:10.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:10.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:10.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:10.530+0000 7f3c1c80d640 -1 Falling back to public interface 2024-04-01T04:59:11.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:11.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:11.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:59:11.517 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:59:11.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:11.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:11.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:12.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:12.803+0000 7f3c1c80d640 -1 osd.0 0 log_to_monitors true 2024-04-01T04:59:12.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:12.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:12.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:59:12.810 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:59:12.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:12.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:13.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:14.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:14.214 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:59:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:59:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:14.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:15.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:15.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:15.513 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T04:59:15.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T04:59:15.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:15.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:15.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:16.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:16.809 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T04:59:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T04:59:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:16.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:17.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:18.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:18.103 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T04:59:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T04:59:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:18.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:19.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:19.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:19.398 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T04:59:19.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T04:59:19.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:19.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:19.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:20.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:20.691 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T04:59:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T04:59:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:20.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:21.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:21.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:21.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T04:59:21.991 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T04:59:21.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:21.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:22.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:23.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:23.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:23.333 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T04:59:23.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T04:59:23.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:23.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:23.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:24.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:24.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:24.873 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T04:59:24.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T04:59:24.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:24.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:25.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:26.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:26.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:26.510 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T04:59:26.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T04:59:26.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:26.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:26.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:27.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:27.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:27.956 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T04:59:27.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T04:59:27.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:27.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:28.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:29.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:29.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:29.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T04:59:29.605 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T04:59:29.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:29.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:29.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:31.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:31.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:31.001 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T04:59:31.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T04:59:31.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:31.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:31.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:32.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:32.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:32.477 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T04:59:32.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T04:59:32.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:32.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:32.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:33.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:33.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:33.802 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T04:59:33.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T04:59:33.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:33.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T04:59:34.098 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2660964904,v1:127.0.0.1:6803/2660964904] [v2:127.0.0.1:6804/2660964904,v1:127.0.0.1:6805/2660964904] exists,up 4e3a9535-7904-40b3-aecd-ec0037c569de 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T04:59:34.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T04:59:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T04:59:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f268948d-7b26-4522-90a2-38b2dc351cc0 2024-04-01T04:59:34.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f268948d-7b26-4522-90a2-38b2dc351cc0' 2024-04-01T04:59:34.105 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 f268948d-7b26-4522-90a2-38b2dc351cc0 2024-04-01T04:59:34.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T04:59:34.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2PwpmipEMBxAAAYX1Ca9Lw1gYM8DHtexZUw== 2024-04-01T04:59:34.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2PwpmipEMBxAAAYX1Ca9Lw1gYM8DHtexZUw=="}' 2024-04-01T04:59:34.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f268948d-7b26-4522-90a2-38b2dc351cc0 -i td/osd-scrub-repair/1/new.json 2024-04-01T04:59:34.454 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:59:34.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T04:59:34.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQA2PwpmipEMBxAAAYX1Ca9Lw1gYM8DHtexZUw== --osd-uuid f268948d-7b26-4522-90a2-38b2dc351cc0 2024-04-01T04:59:34.496 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.496+0000 7fa6b998a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:34.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.538+0000 7fa6b998a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:34.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.540+0000 7fa6b998a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:34.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.540+0000 7fa6b998a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T04:59:34.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.540+0000 7fa6b998a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T04:59:34.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.540+0000 7fa6b998a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T04:59:34.614 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:34.613+0000 7fa6b998a640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T04:59:40.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T04:59:40.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T04:59:40.301 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T04:59:40.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T04:59:40.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T04:59:40.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T04:59:40.667 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T04:59:40.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-04-01T04:59:40.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T04:59:40.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T04:59:40.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T04:59:40.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:40.694+0000 7fc23ae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:40.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:40.696+0000 7fc23ae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:40.697 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:40.697+0000 7fc23ae0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T04:59:40.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T04:59:40.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T04:59:40.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T04:59:40.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T04:59:40.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T04:59:40.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:40.950 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T04:59:40.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T04:59:40.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:40.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:41.226 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:41.226+0000 7fc23ae0d640 -1 Falling back to public interface 2024-04-01T04:59:41.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:42.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:42.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:42.233 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T04:59:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T04:59:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:42.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:42.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:43.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T04:59:43.514+0000 7fc23ae0d640 -1 osd.1 0 log_to_monitors true 2024-04-01T04:59:43.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:43.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:43.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T04:59:43.529 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T04:59:43.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:43.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:43.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:44.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:44.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:44.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T04:59:44.831 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T04:59:44.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:44.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:45.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:46.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:46.262 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T04:59:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T04:59:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:46.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:46.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:47.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:47.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:47.559 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T04:59:47.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T04:59:47.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:47.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:47.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:48.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:48.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:48.864 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T04:59:48.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T04:59:48.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:48.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:49.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:50.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:50.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:50.282 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T04:59:50.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T04:59:50.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:50.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:50.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:51.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:51.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:51.577 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T04:59:51.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T04:59:51.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:51.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:51.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T04:59:52.919 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T04:59:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:54.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:54.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:54.211 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T04:59:54.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T04:59:54.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:54.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:54.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:55.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:55.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:55.504 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T04:59:55.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T04:59:55.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:55.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:57.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:57.027 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T04:59:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T04:59:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:57.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:57.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T04:59:58.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T04:59:58.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T04:59:58.662 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T04:59:58.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T04:59:58.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T04:59:58.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T04:59:59.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:00.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:00.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:00.063 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:00:00.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:00:00.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:00.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:00.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:01.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:01.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:01.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:00:01.882 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:02.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:03.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:03.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:03.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:00:03.293 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:00:03.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:03.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:03.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:04.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:04.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:04.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:00:04.857 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:00:04.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:04.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:05.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:06.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:06.451 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:00:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:00:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:06.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:06.753 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/543868781,v1:127.0.0.1:6811/543868781] [v2:127.0.0.1:6812/543868781,v1:127.0.0.1:6813/543868781] exists,up f268948d-7b26-4522-90a2-38b2dc351cc0 2024-04-01T05:00:06.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:00:06.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:00:06.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:00:06.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-04-01T05:00:06.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:00:07.028 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:00:07.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:00:07.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:00:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:00:07.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:00:08.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:00:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-04-01T05:00:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:00:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:00:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:00:10.021 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:00:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:00:10.022 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:00:10.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:00:10.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:00:10.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:00:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:00:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:00:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:00:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:00:10.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:00:10.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:00:10.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:00:10.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:00:10.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-04-01T05:00:10.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-04-01T05:00:10.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-04-01T05:00:10.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:00:10.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:00:10.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672963 2024-04-01T05:00:10.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672963 2024-04-01T05:00:10.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672963' 2024-04-01T05:00:10.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:00:10.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-04-01T05:00:10.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:00:10.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:00:10.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-04-01T05:00:10.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:00:10.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-04-01T05:00:10.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-04-01T05:00:10.615 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836494 2024-04-01T05:00:10.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:00:10.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-04-01T05:00:10.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:00:11.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:00:11.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:00:12.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-04-01T05:00:12.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:00:12.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672963 2024-04-01T05:00:12.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:00:12.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:00:12.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672963 2024-04-01T05:00:12.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:00:12.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672963 2024-04-01T05:00:12.194 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672963 2024-04-01T05:00:12.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-04-01T05:00:12.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:00:12.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672963 2024-04-01T05:00:12.516 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:00:12.516 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:12.516 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:12.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:00:12.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:00:12.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:00:12.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:00:12.877 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:00:12.877 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:00:12.877 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:00:12.877 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:00:13.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:00:13.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:00:13.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:13.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:13.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:00:13.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:00:13.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:00:13.943 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:00:13.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:00:14.395 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:00:14.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:00:14.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:00:14.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-04-01T05:00:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:00:14.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:00:14.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-04-01T05:00:14.910 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:00:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:00:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:00:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:00:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-04-01T05:00:16.767 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-04-01T05:00:17.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:00:17.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:00:17.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:00:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:00:17.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:00:17.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:00:17.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:00:17.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:00:17.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:00:17.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:17.314+0000 7fcea67df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:17.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:17.316+0000 7fcea67df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:17.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:17.317+0000 7fcea67df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:17.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:00:17.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:17.833+0000 7fcea67df640 -1 Falling back to public interface 2024-04-01T05:00:17.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:18.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:18.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:18.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:00:18.868 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:00:18.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:18.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:00:19.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:00:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:00:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:00:20.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:20.258+0000 7fcea67df640 -1 osd.0 22 log_to_monitors true 2024-04-01T05:00:20.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:21.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:21.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:21.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:00:21.479 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:00:21.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:21.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:00:21.805 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/1353108769,v1:127.0.0.1:6803/1353108769] [v2:127.0.0.1:6804/1353108769,v1:127.0.0.1:6805/1353108769] exists,up 4e3a9535-7904-40b3-aecd-ec0037c569de 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:00:21.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:00:21.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:00:21.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:00:21.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:00:21.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:00:21.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:00:21.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:00:21.888 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:00:22.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:00:22.173 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:00:22.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:00:22.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:00:22.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:00:22.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-04-01T05:00:22.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-04-01T05:00:22.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-04-01T05:00:22.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:00:22.278 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:00:22.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-04-01T05:00:22.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-04-01T05:00:22.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672967' 2024-04-01T05:00:22.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:00:22.383 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-04-01T05:00:22.383 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:00:22.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:00:22.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-04-01T05:00:22.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:00:22.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-04-01T05:00:22.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-04-01T05:00:22.385 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 111669149698 2024-04-01T05:00:22.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:00:22.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-04-01T05:00:22.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:00:23.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:00:23.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:00:24.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-04-01T05:00:24.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:00:24.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-04-01T05:00:24.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:00:24.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:00:24.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-04-01T05:00:24.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:00:24.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-04-01T05:00:24.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-04-01T05:00:24.002 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672967 2024-04-01T05:00:24.003 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:00:24.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-04-01T05:00:24.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:00:24.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:24.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:24.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:00:24.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:00:24.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:00:24.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:00:24.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:24.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:25.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-04-01T05:00:25.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:00:25.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-04-01T05:00:25.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-04-01T05:00:25.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-04-01T05:00:25.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-04-01T05:00:25.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:25.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:25.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:25.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:25.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:25.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-04-01T05:00:26.109 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 1.3 on osd.1 to repair 2024-04-01T05:00:26.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:26.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-04-01T05:00:26.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:26.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:26.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:00:07.367373+0000 '>' 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:26.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:00:27.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:00:27.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:00:27.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:00:27.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:27.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:27.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:27.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:27.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:00:07.367373+0000 '>' 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:27.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:00:28.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:00:28.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:00:28.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:00:28.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:28.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:28.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:28.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:29.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:00:07.367373+0000 '>' 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:29.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:00:30.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:00:30.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:00:30.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:00:30.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:30.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:30.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:30.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:30.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:00:07.367373+0000 '>' 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:30.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:00:31.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:00:31.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:00:31.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:00:31.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:31.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:31.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:31.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:31.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:00:07.367373+0000 '>' 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:31.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:00:32.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:00:32.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:00:32.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:00:32.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:00:32.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:00:32.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:00:32.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:00:26.413823+0000 '>' 2024-04-01T05:00:07.367373+0000 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:00:32.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:00:32.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:00:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:00:34.362 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:00:34.362 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:00:34.363 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:00:34.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:00:34.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:00:34.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:00:34.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:00:34.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:00:34.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:00:34.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:00:34.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:00:34.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:00:34.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:00:34.635 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:00:34.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:00:34.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:00:34.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:00:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:00:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:00:34.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:00:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:34.658+0000 7f3ad0e9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:34.661+0000 7f3ad0e9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:34.662+0000 7f3ad0e9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:35.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:35.177+0000 7f3ad0e9b640 -1 Falling back to public interface 2024-04-01T05:00:35.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:36.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:36.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:36.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:00:36.209 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:00:36.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:36.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:36.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:00:37.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:37.358+0000 7f3ad0e9b640 -1 osd.1 28 log_to_monitors true 2024-04-01T05:00:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:00:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:00:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:00:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:00:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:00:37.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:00:37.926 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 32 up_thru 27 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3034636096,v1:127.0.0.1:6811/3034636096] [v2:127.0.0.1:6812/3034636096,v1:127.0.0.1:6813/3034636096] exists,up f268948d-7b26-4522-90a2-38b2dc351cc0 2024-04-01T05:00:37.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:00:37.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:00:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:00:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:00:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:00:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:00:38.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:00:38.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:00:38.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:00:38.294 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:00:38.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:00:38.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:00:38.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:00:38.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-04-01T05:00:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-04-01T05:00:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-04-01T05:00:38.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:00:38.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:00:38.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-04-01T05:00:38.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-04-01T05:00:38.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-04-01T05:00:38.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:00:38.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-04-01T05:00:38.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:00:38.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:00:38.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-04-01T05:00:38.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:00:38.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-04-01T05:00:38.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-04-01T05:00:38.582 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 111669149702 2024-04-01T05:00:38.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:00:38.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-04-01T05:00:38.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:00:39.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:00:39.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:00:40.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-04-01T05:00:40.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:00:40.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-04-01T05:00:40.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:00:40.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:00:40.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-04-01T05:00:40.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:00:40.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-04-01T05:00:40.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-04-01T05:00:40.221 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 137438953474 2024-04-01T05:00:40.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:00:40.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-04-01T05:00:40.503 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:00:40.503 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:40.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:00:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:00:41.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-01T05:00:41.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:00:41.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:41.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:41.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-04-01T05:00:41.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-04-01T05:00:41.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:00:41.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-01T05:00:41.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:00:41.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:00:41.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:00:41.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:00:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:00:41.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:00:41.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:00:41.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:00:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:00:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:00:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:00:42.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:00:42.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:00:42.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:00:42.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:00:42.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:00:42.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:00:42.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:00:42.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:00:42.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:00:42.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:00:42.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:00:42.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:00:42.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:00:42.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:00:42.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:00:42.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:00:42.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:00:42.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:00:42.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:00:42.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:00:42.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:00:42.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:00:42.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:00:42.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:00:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:00:42.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:00:42.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:00:42.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:00:42.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:00:42.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:00:42.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:00:42.382 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:00:42.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:00:42.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:00:42.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:00:42.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:00:42.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:00:42.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:00:42.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:00:42.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:00:42.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:00:42.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:00:42.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:00:42.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:00:42.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:00:42.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:00:42.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:00:42.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:00:42.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:00:42.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:00:42.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-04-01T05:00:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-04-01T05:00:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-04-01T05:00:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-04-01T05:00:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-04-01T05:00:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-04-01T05:00:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-04-01T05:00:43.913 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 dba8f4a3-4e16-438a-a708-500c92c344b6 2024-04-01T05:00:44.144 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:00:44.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.187+0000 7fcaf7a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:44.195 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.194+0000 7fcaf7a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:44.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.195+0000 7fcaf7a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:44.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.196+0000 7fcaf7a24640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:00:44.196 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.196+0000 7fcaf7a24640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:00:44.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.196+0000 7fcaf7a24640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:00:44.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:44.278+0000 7fcaf7a24640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:00:49.743 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:00:50.119 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:00:50.150 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:50.149+0000 7f9b53062640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:50.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:50.159+0000 7f9b53062640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:50.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:50.160+0000 7f9b53062640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:00:50.436 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:00:50.690 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:50.689+0000 7f9b53062640 -1 Falling back to public interface 2024-04-01T05:00:51.715 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:00:52.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:00:52.976+0000 7f9b53062640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:00:53.010 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:00:54.362 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:00:55.687 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:00:56.983 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:00:58.275 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:00:59.568 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:01:00.863 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:01:02.160 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:01:03.469 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:01:05.007 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:01:06.602 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:01:08.090 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:01:09.791 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:01:11.185 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:01:12.997 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:01:14.783 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:01:15.081 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2211336339,v1:127.0.0.1:6803/2211336339] [v2:127.0.0.1:6804/2211336339,v1:127.0.0.1:6805/2211336339] exists,up dba8f4a3-4e16-438a-a708-500c92c344b6 2024-04-01T05:01:15.084 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 6cc8f522-ecb7-4d8b-b400-5f3f917fe4b0 2024-04-01T05:01:15.537 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:01:15.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.582+0000 7f8d26010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:15.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.621+0000 7f8d26010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:15.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.622+0000 7f8d26010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:15.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.623+0000 7f8d26010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:01:15.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.623+0000 7f8d26010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:01:15.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.623+0000 7f8d26010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:01:15.688 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:15.688+0000 7f8d26010640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:01:21.165 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:01:21.533 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:01:21.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:21.565+0000 7f2719b73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:21.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:21.567+0000 7f2719b73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:21.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:21.568+0000 7f2719b73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:21.814 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:01:22.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:22.093+0000 7f2719b73640 -1 Falling back to public interface 2024-04-01T05:01:23.099 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:01:24.390 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:01:24.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:24.437+0000 7f2719b73640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:01:25.686 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:01:26.980 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:01:28.285 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:01:29.581 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:01:30.871 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:01:32.251 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:01:33.551 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:01:34.847 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:01:36.177 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:01:37.947 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:01:39.884 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:01:41.535 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:01:43.008 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:01:44.692 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:01:46.286 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:01:46.580 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/215936488,v1:127.0.0.1:6811/215936488] [v2:127.0.0.1:6812/215936488,v1:127.0.0.1:6813/215936488] exists,up 6cc8f522-ecb7-4d8b-b400-5f3f917fe4b0 2024-04-01T05:01:46.584 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 9e0ac62f-e7f2-45da-af7f-ed5f04087fbe 2024-04-01T05:01:46.936 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:01:46.981 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:46.980+0000 7f3126a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:47.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:47.012+0000 7f3126a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:47.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:47.013+0000 7f3126a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:47.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:47.013+0000 7f3126a0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:01:47.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:47.013+0000 7f3126a0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:01:47.014 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:47.013+0000 7f3126a0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:01:47.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:47.087+0000 7f3126a0f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:01:53.049 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:01:53.415 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:01:53.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:53.446+0000 7f3949f2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:53.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:53.448+0000 7f3949f2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:53.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:53.449+0000 7f3949f2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:01:53.697 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:01:53.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:53.970+0000 7f3949f2a640 -1 Falling back to public interface 2024-04-01T05:01:54.985 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:01:56.276 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:01:56.321 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:01:56.320+0000 7f3949f2a640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:01:57.655 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:01:58.952 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:02:00.322 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:02:01.619 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:02:02.937 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:02:04.313 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:02:05.606 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:02:06.921 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:02:08.512 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:02:10.464 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:02:12.241 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:02:13.928 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:02:15.389 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:02:17.058 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:02:18.725 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:02:19.029 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/41384356,v1:127.0.0.1:6819/41384356] [v2:127.0.0.1:6820/41384356,v1:127.0.0.1:6821/41384356] exists,up 9e0ac62f-e7f2-45da-af7f-ed5f04087fbe 2024-04-01T05:02:19.407 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T05:02:21.121 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836495 2024-04-01T05:02:22.705 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672973 2024-04-01T05:02:23.016 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509446 2024-04-01T05:02:25.578 INFO:tasks.workunit.client.0.smithi067.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-04-01T05:02:25.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-04-01T05:02:25.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-04-01T05:02:25.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-04-01T05:02:25.950 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 2 2024-04-01T05:02:25.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-04-01T05:02:25.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:02:25.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:02:25.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:02:25.962 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:02:25.962 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:02:25.963 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:02:25.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:02:25.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:02:25.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:02:26.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:02:26.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:02:26.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:02:26.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:02:26.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:02:26.046 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:02:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:02:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:02:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:02:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:02:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:26.491 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:02:26.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-04-01T05:02:26.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-04-01T05:02:26.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-04-01T05:02:26.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:26.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:02:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-04-01T05:02:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-04-01T05:02:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672975' 2024-04-01T05:02:26.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:26.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:02:26.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-04-01T05:02:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-04-01T05:02:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672975 2-64424509448' 2024-04-01T05:02:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:26.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-04-01T05:02:26.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:02:26.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-04-01T05:02:26.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-04-01T05:02:26.848 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836497 2024-04-01T05:02:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-04-01T05:02:26.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:27.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-04-01T05:02:27.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:02:28.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:02:28.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:28.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-04-01T05:02:28.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:28.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-04-01T05:02:28.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:28.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:02:28.442 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-04-01T05:02:28.442 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:28.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-04-01T05:02:28.444 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672975 2024-04-01T05:02:28.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-01T05:02:28.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:02:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-01T05:02:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:28.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-04-01T05:02:28.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:28.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:02:28.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-04-01T05:02:28.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-04-01T05:02:28.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-01T05:02:28.727 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509448 2024-04-01T05:02:28.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:02:29.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-04-01T05:02:29.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:02:29.013 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:29.013 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:02:29.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:02:29.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:02:29.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:02:29.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:02:29.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:29.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:30.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:02:30.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:02:30.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:02:30.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-04-01T05:02:30.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-04-01T05:02:30.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:02:30.216 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:02:30.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:02:30.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:02:30.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:02:30.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:02:30.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:02:30.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:02:30.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:02:30.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:02:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:02:31.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-04-01T05:02:32.340 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-04-01T05:02:32.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:02:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:02:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:02:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:02:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:02:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:02:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:02:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:02:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:02:32.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:02:32.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:02:32.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:02:32.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:02:32.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:02:32.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:02:32.889 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:32.888+0000 7fa2b7147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:02:32.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:32.890+0000 7fa2b7147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:02:32.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:32.892+0000 7fa2b7147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:33.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:33.406+0000 7fa2b7147640 -1 Falling back to public interface 2024-04-01T05:02:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:02:34.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:02:34.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:02:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:02:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:02:35.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:02:35.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:35.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:02:35.730 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:02:35.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:35.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:35.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:35.738+0000 7fa2b7147640 -1 osd.0 21 log_to_monitors true 2024-04-01T05:02:36.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:02:37.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:02:37.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:37.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:02:37.036 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:02:37.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:37.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1628812743,v1:127.0.0.1:6803/1628812743] [v2:127.0.0.1:6804/1628812743,v1:127.0.0.1:6805/1628812743] exists,up dba8f4a3-4e16-438a-a708-500c92c344b6 2024-04-01T05:02:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:02:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:02:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:02:37.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:02:37.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:02:37.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:02:37.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:02:37.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:02:37.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:02:37.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:02:37.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:02:37.744 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:02:37.745 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:02:37.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:02:37.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:37.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:02:37.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-04-01T05:02:37.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-04-01T05:02:37.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-04-01T05:02:37.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:37.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:02:37.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-04-01T05:02:37.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-04-01T05:02:37.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-04-01T05:02:37.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:37.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:02:38.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-04-01T05:02:38.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-04-01T05:02:38.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509451' 2024-04-01T05:02:38.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:38.061 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-04-01T05:02:38.061 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:38.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:02:38.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-04-01T05:02:38.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:38.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-04-01T05:02:38.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-04-01T05:02:38.063 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182402 2024-04-01T05:02:38.063 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:38.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-04-01T05:02:38.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:38.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-04-01T05:02:38.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:38.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:02:38.353 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-04-01T05:02:38.353 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:38.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-04-01T05:02:38.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-01T05:02:38.354 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672978 2024-04-01T05:02:38.354 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:02:38.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-01T05:02:38.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:38.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-04-01T05:02:38.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:38.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:02:38.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-04-01T05:02:38.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:38.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-04-01T05:02:38.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-01T05:02:38.643 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509451 2024-04-01T05:02:38.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:02:38.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509451 2024-04-01T05:02:38.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:02:39.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:02:39.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:02:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-04-01T05:02:40.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:02:40.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:40.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:40.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:02:40.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:02:40.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:02:40.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:02:40.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:40.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:02:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:02:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:02:41.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-04-01T05:02:41.311 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0: { 2024-04-01T05:02:41.311 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-04-01T05:02:41.311 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:02:41.325 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1: { 2024-04-01T05:02:41.325 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-04-01T05:02:41.326 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:02:41.339 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2: { 2024-04-01T05:02:41.339 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-04-01T05:02:41.339 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:02:41.349 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-04-01T05:02:41.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T05:02:41.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:02:41.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-04-01T05:02:41.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:02:41.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-04-01T05:02:41.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-04-01T05:02:41.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:02:41.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:02:41.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:02:41.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:02:41.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-04-01T05:02:41.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-04-01T05:02:41.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:02:41.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:02:41.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:02:41.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:02:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:42.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-04-01T05:02:42.298 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T05:02:42.298 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": true, 2024-04-01T05:02:42.298 INFO:tasks.workunit.client.0.smithi067.stdout: "must": false, 2024-04-01T05:02:42.298 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "2024-03-18T05:01:02.298590+0000" 2024-04-01T05:02:42.298 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:02:42.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:02:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:02:19.356347+0000 '>' 2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:02:43.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:02:43.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:02:43.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:02:43.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:02:43.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:02:43.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:02:43.587 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:02:43.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:02:19.356347+0000 '>' 2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:43.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:02:44.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:02:44.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:02:44.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:02:44.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:02:44.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:02:44.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:02:44.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:02:45.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:02:19.356347+0000 '>' 2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:45.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:02:46.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:02:46.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:02:46.169 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:02:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:02:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:02:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:02:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:02:46.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:02:42.630357+0000 '>' 2024-04-01T05:02:19.356347+0000 2024-04-01T05:02:46.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:02:46.462 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:02:46.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:02:46.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:02:46.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:02:46.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:02:46.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:02:46.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:02:46.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:02:46.831 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:02:46.831 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:02:46.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:02:46.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:46.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:02:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-04-01T05:02:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-04-01T05:02:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-04-01T05:02:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:46.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:02:47.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-04-01T05:02:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-04-01T05:02:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672981' 2024-04-01T05:02:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:47.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:02:47.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-04-01T05:02:47.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-04-01T05:02:47.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672981 2-64424509454' 2024-04-01T05:02:47.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:47.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-04-01T05:02:47.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:47.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:02:47.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-04-01T05:02:47.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:47.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-04-01T05:02:47.149 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 107374182405 2024-04-01T05:02:47.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-04-01T05:02:47.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:47.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-04-01T05:02:47.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:02:48.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:02:48.438 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:48.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-04-01T05:02:48.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:48.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-04-01T05:02:48.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:48.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:02:48.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-04-01T05:02:48.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:48.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-04-01T05:02:48.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-01T05:02:48.735 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672981 2024-04-01T05:02:48.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:02:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-04-01T05:02:49.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:49.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-04-01T05:02:49.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:49.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:02:49.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-04-01T05:02:49.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:49.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-04-01T05:02:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-01T05:02:49.044 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509454 2024-04-01T05:02:49.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:02:49.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-01T05:02:49.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:02:49.326 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:49.326 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:49.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:02:49.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:02:49.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:02:49.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:02:49.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:02:49.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:02:49.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:02:49.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:02:49.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:02:49.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:02:49.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:49.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:50.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:02:50.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:02:50.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:02:50.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-04-01T05:02:50.619 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:02:50.620 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-04-01T05:02:42.804905+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-04-01T05:02:42.630357+0000 21'1 2024-04-01T05:02:42.630357+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:02:42.630357+0000 1 0 2024-04-01T05:02:50.620 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:02:50.620 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:02:50.620 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:02:50.631 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:02:50.632 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:02:50.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:02:50.916 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:02:50.916 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:02:51.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:02:51.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:02:51.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:02:51.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:02:51.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:02:51.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:02:51.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:02:51.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:02:51.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:02:51.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:02:51.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:02:51.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-04-01T05:02:52.737 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:02:52.737 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:02:53.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:02:53.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:02:53.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:02:53.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:02:53.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:02:53.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:02:53.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:02:53.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:02:53.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:02:53.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:53.037+0000 7f5ca9cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:02:53.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:53.076+0000 7f5ca9cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:02:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:53.077+0000 7f5ca9cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:02:53.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:53.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:53.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:02:53.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:53.599+0000 7f5ca9cc8640 -1 Falling back to public interface 2024-04-01T05:02:54.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:02:54.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:54.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:02:54.585 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:02:54.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:54.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:02:55.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:02:55.804+0000 7f5ca9cc8640 -1 osd.0 26 log_to_monitors true 2024-04-01T05:02:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:02:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:02:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:02:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:02:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:02:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:02:56.209 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2608067492,v1:127.0.0.1:6803/2608067492] [v2:127.0.0.1:6804/2608067492,v1:127.0.0.1:6805/2608067492] exists,up dba8f4a3-4e16-438a-a708-500c92c344b6 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:02:56.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:02:56.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:02:56.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:02:56.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:02:56.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:02:56.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:02:56.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:02:56.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:02:56.579 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:02:56.579 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:02:56.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:02:56.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:56.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:02:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-04-01T05:02:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-04-01T05:02:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-04-01T05:02:56.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:56.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:02:56.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-04-01T05:02:56.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-04-01T05:02:56.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672984' 2024-04-01T05:02:56.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:02:56.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:02:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-04-01T05:02:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-04-01T05:02:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672984 2-64424509457' 2024-04-01T05:02:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:56.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-04-01T05:02:56.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:56.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:02:56.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-04-01T05:02:56.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:56.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-04-01T05:02:56.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-04-01T05:02:56.922 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 128849018882 2024-04-01T05:02:56.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-04-01T05:02:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:02:58.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:02:58.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:02:58.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-04-01T05:02:58.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:58.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-04-01T05:02:58.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:58.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:02:58.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-04-01T05:02:58.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:58.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-04-01T05:02:58.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-01T05:02:58.527 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672984 2024-04-01T05:02:58.527 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:02:58.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-01T05:02:58.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:02:58.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-04-01T05:02:58.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:02:58.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:02:58.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-04-01T05:02:58.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:02:58.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-04-01T05:02:58.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-01T05:02:58.846 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509457 2024-04-01T05:02:58.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:02:59.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-04-01T05:02:59.132 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:02:59.132 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:59.132 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:02:59.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:02:59.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:02:59.772 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:02:59.772 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:02:59.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:03:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:03:00.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-01T05:03:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:03:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-01T05:03:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:03:00.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:03:00.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:03:00.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:03:00.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:03:00.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:03:00.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:03:00.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:03:00.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-01T05:03:00.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:03:00.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:03:00.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:03:00.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:03:00.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:03:00.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:03:00.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:03:01.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:03:01.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:03:01.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:03:01.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:03:01.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:03:01.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:03:01.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:03:01.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:03:01.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:03:01.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:03:01.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:03:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:03:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:03:02.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:03:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:03:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:03:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-01T05:03:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:03:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:03:02.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:03:03.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:03:03.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:03:03.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:03:03.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:03:03.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:03:03.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:03:03.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:03:03.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:03:03.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:03:04.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:03:04.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:03:04.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:03:04.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:03:04.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:03:05.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:03:05.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:03:05.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:03:05.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:03:05.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:03:05.761 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:03:05.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:03:05.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:03:05.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:03:05.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:03:05.787 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:05.785+0000 7fb84ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:05.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:05.788+0000 7fb84ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:05.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:05.789+0000 7fb84ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:06.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:03:06.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:03:06.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:06.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:06.307+0000 7fb84ca0d640 -1 Falling back to public interface 2024-04-01T05:03:06.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:07.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:07.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:07.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:03:07.338 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:03:07.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:07.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:07.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:08.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:08.594+0000 7fb84ca0d640 -1 osd.0 31 log_to_monitors true 2024-04-01T05:03:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:03:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:03:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:09.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:03:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:03:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:10.378 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/203573520,v1:127.0.0.1:6803/203573520] [v2:127.0.0.1:6804/203573520,v1:127.0.0.1:6805/203573520] exists,up dba8f4a3-4e16-438a-a708-500c92c344b6 2024-04-01T05:03:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:03:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:03:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:03:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:03:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:03:10.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:03:10.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:03:10.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:03:10.380 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:03:10.380 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:03:10.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:03:10.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:03:10.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:03:10.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:03:10.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:03:10.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:03:10.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:03:10.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:03:10.461 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:03:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:03:10.740 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:03:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:03:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:03:10.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:03:10.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:03:10.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-04-01T05:03:10.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-04-01T05:03:10.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-04-01T05:03:10.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:03:10.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:03:10.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-04-01T05:03:10.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-04-01T05:03:10.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672987' 2024-04-01T05:03:10.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:03:10.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:03:11.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-04-01T05:03:11.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-04-01T05:03:11.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672987 2-64424509461' 2024-04-01T05:03:11.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:03:11.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-04-01T05:03:11.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:03:11.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:03:11.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-04-01T05:03:11.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:03:11.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-04-01T05:03:11.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-04-01T05:03:11.062 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 150323855362 2024-04-01T05:03:11.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:03:11.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-04-01T05:03:11.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:03:12.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:03:12.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:03:12.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-04-01T05:03:12.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:03:12.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-04-01T05:03:12.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:03:12.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:03:12.643 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-04-01T05:03:12.643 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:03:12.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-04-01T05:03:12.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-04-01T05:03:12.645 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672987 2024-04-01T05:03:12.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:03:12.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672987 2024-04-01T05:03:12.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:03:12.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-04-01T05:03:12.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:03:12.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:03:12.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-04-01T05:03:12.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:03:12.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-04-01T05:03:12.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-04-01T05:03:12.930 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509461 2024-04-01T05:03:12.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:03:13.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-04-01T05:03:13.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:03:13.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:03:13.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:03:13.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:03:13.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:03:13.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:03:13.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:03:13.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:03:13.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:03:13.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:03:13.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:03:13.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:03:13.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:03:13.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:03:13.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:03:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:03:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:03:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:03:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:03:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-04-01T05:03:14.247 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:02:42.629+0000 7f26fdaec640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:03:14.248 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:02:42.629+0000 7f26fdaec640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:03:14.248 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:02:42.629+0000 7f26fdaec640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-04-01T05:03:14.248 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:02:42.629+0000 7f26fdaec640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-04-01T05:03:14.248 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:02:42.629+0000 7f26fdaec640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:03:14.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:03:14.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:03:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:03:14.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:03:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:03:14.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:03:14.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:03:14.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:03:14.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:03:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:03:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:03:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:03:14.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:03:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:03:14.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:03:14.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:03:14.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:03:14.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:14.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:14.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:03:14.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:03:14.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:03:14.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:03:14.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:03:14.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:03:14.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:03:14.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:03:14.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:03:14.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:03:14.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:03:14.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:03:14.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:03:14.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:03:14.383 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:03:14.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:03:14.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:03:14.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:03:14.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:03:14.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:14.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:14.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:03:14.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:03:14.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:03:14.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:03:14.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:03:14.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:14.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:14.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:03:14.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:03:14.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:03:14.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:14.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:03:14.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:03:14.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:03:14.948 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:03:15.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:15.028 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:03:15.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:03:15.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:03:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-04-01T05:03:15.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:03:15.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:03:15.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:03:15.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:03:15.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:03:15.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:03:15.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:03:15.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:03:15.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:03:15.905 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:03:15.906 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:15.906 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:15.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:15.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:03:15.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:03:15.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:03:15.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:03:15.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:03:15.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:03:15.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:03:15.939 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:03:15.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 36d59fbc-bfc7-4330-895f-6863e02fe98b' 2024-04-01T05:03:15.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:03:15.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATQApmb+3ZOBAAiVETPz8NHEfhvn49OcUQIQ== 2024-04-01T05:03:15.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATQApmb+3ZOBAAiVETPz8NHEfhvn49OcUQIQ=="}' 2024-04-01T05:03:15.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36d59fbc-bfc7-4330-895f-6863e02fe98b -i td/osd-scrub-repair/0/new.json 2024-04-01T05:03:16.186 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:03:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:03:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQATQApmb+3ZOBAAiVETPz8NHEfhvn49OcUQIQ== --osd-uuid 36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:03:16.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.227+0000 7f14969ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:16.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.237+0000 7f14969ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:16.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.238+0000 7f14969ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:16.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.238+0000 7f14969ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:03:16.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.238+0000 7f14969ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:03:16.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.238+0000 7f14969ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:03:16.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:16.329+0000 7f14969ac640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:03:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:03:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:03:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:03:21.696 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:03:21.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:03:22.065 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:03:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:03:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:03:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:03:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:03:22.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:03:22.095 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:22.094+0000 7f12d99fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:22.096 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:22.096+0000 7f12d99fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:22.097 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:22.097+0000 7f12d99fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:22.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:03:22.350 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:03:22.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:22.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:22.610 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:22.610+0000 7f12d99fd640 -1 Falling back to public interface 2024-04-01T05:03:22.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:23.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:23.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:23.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:03:23.632 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:03:23.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:23.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:23.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:24.868 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:24.868+0000 7f12d99fd640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:03:24.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:24.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:24.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:03:24.924 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:03:24.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:24.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:25.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:26.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:26.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:26.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:03:26.278 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:03:26.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:26.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:26.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:27.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:27.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:27.577 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:03:27.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:03:27.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:27.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:27.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:28.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:28.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:28.872 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:03:28.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:03:28.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:28.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:29.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:30.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:30.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:30.170 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:03:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:03:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:30.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:31.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:31.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:31.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:03:31.568 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:03:31.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:31.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:32.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:32.860 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:03:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:03:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:32.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:33.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:34.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:34.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:34.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:03:34.157 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:03:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:34.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:34.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:35.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:35.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:35.536 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:03:35.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:03:35.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:35.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:36.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:37.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:37.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:37.072 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:03:37.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:03:37.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:37.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:37.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:38.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:38.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:38.564 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:03:38.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:03:38.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:38.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:39.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:40.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:40.229 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:03:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:03:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:40.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:41.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:42.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:42.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:42.197 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:03:42.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:03:42.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:42.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:42.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:43.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:43.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:43.974 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:03:43.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:03:43.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:43.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:44.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:45.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:45.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:45.339 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:03:45.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:03:45.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:45.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:45.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:03:46.637 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:03:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:46.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:03:46.931 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3532601408,v1:127.0.0.1:6803/3532601408] [v2:127.0.0.1:6804/3532601408,v1:127.0.0.1:6805/3532601408] exists,up 36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:03:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:03:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:03:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:03:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-04-01T05:03:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:03:46.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:03:46.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:03:46.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d' 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d 2024-04-01T05:03:46.938 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:03:46.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyQApmRIi5OBAAwv2aRar65PJ0ylwaALgAmw== 2024-04-01T05:03:46.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyQApmRIi5OBAAwv2aRar65PJ0ylwaALgAmw=="}' 2024-04-01T05:03:46.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d -i td/osd-scrub-repair/1/new.json 2024-04-01T05:03:47.348 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:03:47.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:03:47.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAyQApmRIi5OBAAwv2aRar65PJ0ylwaALgAmw== --osd-uuid 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d 2024-04-01T05:03:47.391 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.389+0000 7f0b951bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:47.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.431+0000 7f0b951bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:47.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.432+0000 7f0b951bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:47.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.432+0000 7f0b951bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:03:47.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.432+0000 7f0b951bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:03:47.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.432+0000 7f0b951bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:03:47.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:47.497+0000 7f0b951bd640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:03:52.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:03:52.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:03:52.972 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:03:52.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:03:52.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:03:53.346 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:03:53.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:03:53.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:03:53.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:03:53.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:03:53.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:03:53.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:53.374+0000 7f5aa2611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:53.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:53.376+0000 7f5aa2611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:53.377 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:53.377+0000 7f5aa2611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:53.631 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:03:53.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:03:53.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:53.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:03:53.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:53.903+0000 7f5aa2611640 -1 Falling back to public interface 2024-04-01T05:03:53.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:54.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:54.916 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:03:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:03:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:03:55.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:56.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:56.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:56.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:03:56.222 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:03:56.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:56.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:03:56.289 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:03:56.289+0000 7f5aa2611640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:03:56.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:57.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:57.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:57.516 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:03:57.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:03:57.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:57.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:03:57.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:03:58.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:03:58.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:03:58.863 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:03:58.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:03:58.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:03:58.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:03:59.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:00.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:00.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:00.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:04:00.354 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:04:00.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:00.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:01.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:01.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:01.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:04:01.648 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:04:01.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:01.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:02.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:03.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:04:03.033 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:04:03.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:03.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:03.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:04.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:04.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:04.380 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:04:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:04:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:05.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:05.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:05.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:04:05.672 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:04:05.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:05.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:06.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:07.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:07.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:07.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:04:07.033 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:04:07.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:07.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:07.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:08.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:08.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:08.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:04:08.326 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:04:08.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:08.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:09.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:10.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:10.008 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:04:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:04:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:10.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:10.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:11.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:11.643 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:04:11.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:04:11.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:11.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:12.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:13.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:13.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:13.213 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:04:13.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:04:13.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:13.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:13.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:14.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:14.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:14.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:04:14.825 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:04:14.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:14.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:15.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:16.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:16.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:16.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:04:16.553 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:04:16.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:16.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:17.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:18.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:18.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:18.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:04:18.322 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:04:18.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:18.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:04:18.693 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2931565525,v1:127.0.0.1:6811/2931565525] [v2:127.0.0.1:6812/2931565525,v1:127.0.0.1:6813/2931565525] exists,up 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:04:18.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:04:18.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=48c0515b-e097-42f0-a10f-27c38c251d65 2024-04-01T05:04:18.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 48c0515b-e097-42f0-a10f-27c38c251d65' 2024-04-01T05:04:18.700 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 48c0515b-e097-42f0-a10f-27c38c251d65 2024-04-01T05:04:18.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:04:18.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSQApmHHyFKhAAidxm1Dpgd2nNLitq/N9lbg== 2024-04-01T05:04:18.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSQApmHHyFKhAAidxm1Dpgd2nNLitq/N9lbg=="}' 2024-04-01T05:04:18.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 48c0515b-e097-42f0-a10f-27c38c251d65 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:04:19.132 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:04:19.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:04:19.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBSQApmHHyFKhAAidxm1Dpgd2nNLitq/N9lbg== --osd-uuid 48c0515b-e097-42f0-a10f-27c38c251d65 2024-04-01T05:04:19.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.173+0000 7ffacb613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:04:19.197 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.197+0000 7ffacb613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:04:19.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.197+0000 7ffacb613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:04:19.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.198+0000 7ffacb613640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:04:19.198 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.198+0000 7ffacb613640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:04:19.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.198+0000 7ffacb613640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:04:19.258 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:19.258+0000 7ffacb613640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:04:24.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:04:24.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:04:24.973 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:04:24.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:04:24.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:04:25.368 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:04:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:04:25.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:04:25.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:04:25.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:04:25.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:04:25.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:25.395+0000 7fbcb0015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:04:25.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:25.398+0000 7fbcb0015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:04:25.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:25.399+0000 7fbcb0015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:25.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:04:25.655 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:04:25.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:25.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:25.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:25.917+0000 7fbcb0015640 -1 Falling back to public interface 2024-04-01T05:04:25.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:26.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:26.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:26.974 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:04:26.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:04:26.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:26.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:27.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:28.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:04:28.249+0000 7fbcb0015640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:04:28.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:28.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:28.286 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:04:28.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:04:28.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:28.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:28.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:29.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:29.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:29.737 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:04:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:04:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:29.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:31.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:31.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:31.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:04:31.036 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:04:31.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:31.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:31.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:32.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:32.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:32.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:04:32.401 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:04:32.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:32.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:32.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:33.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:33.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:33.699 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:04:33.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:04:33.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:33.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:34.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:35.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:35.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:35.056 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:04:35.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:04:35.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:35.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:35.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:36.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:36.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:36.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:04:36.348 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:04:36.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:36.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:36.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:37.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:37.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:37.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:04:37.642 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:04:37.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:37.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:37.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:04:38.937 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:04:38.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:38.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:39.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:40.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:40.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:40.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:04:40.373 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:04:40.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:40.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:41.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:42.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:42.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:42.198 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:04:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:04:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:42.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:43.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:44.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:44.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:44.102 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:04:44.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:04:44.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:44.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:44.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:45.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:45.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:45.487 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:04:45.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:04:45.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:45.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:46.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:47.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:47.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:47.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:04:47.182 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:04:47.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:47.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:47.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:48.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:48.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:48.611 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:04:48.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:04:48.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:48.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:48.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:49.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:49.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:49.905 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:04:49.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:04:49.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:49.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:50.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:04:51.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:04:51.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:04:51.225 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:04:51.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:04:51.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:04:51.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:04:51.518 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1232907515,v1:127.0.0.1:6819/1232907515] [v2:127.0.0.1:6820/1232907515,v1:127.0.0.1:6821/1232907515] exists,up 48c0515b-e097-42f0-a10f-27c38c251d65 2024-04-01T05:04:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:04:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:04:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:04:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-04-01T05:04:51.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-04-01T05:04:51.927 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T05:04:51.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:04:52.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-04-01T05:04:53.418 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 2 2024-04-01T05:04:53.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-04-01T05:04:53.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:04:53.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:04:53.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:04:53.430 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:04:53.431 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:04:53.431 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:04:53.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:04:53.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:04:53.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:04:53.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:04:53.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:04:53.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:04:53.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:04:53.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:04:53.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:04:53.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:04:53.864 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:04:53.864 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:04:53.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:04:53.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:04:53.865 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:04:54.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-04-01T05:04:54.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-04-01T05:04:54.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-04-01T05:04:54.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:04:54.023 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:04:54.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-04-01T05:04:54.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-04-01T05:04:54.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672970' 2024-04-01T05:04:54.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:04:54.129 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:04:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-04-01T05:04:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-04-01T05:04:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672970 2-64424509449' 2024-04-01T05:04:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:04:54.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-04-01T05:04:54.235 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:04:54.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:04:54.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-04-01T05:04:54.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:04:54.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-04-01T05:04:54.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-01T05:04:54.237 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836498 2024-04-01T05:04:54.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:04:54.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-01T05:04:54.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:04:54.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-04-01T05:04:54.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:04:54.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:04:54.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-04-01T05:04:54.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:04:54.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-04-01T05:04:54.523 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672970 2024-04-01T05:04:54.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-04-01T05:04:54.523 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:04:54.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672970 2024-04-01T05:04:54.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:04:55.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:04:55.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:04:56.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672970 2024-04-01T05:04:56.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:04:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:04:57.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:04:57.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-04-01T05:04:57.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:04:57.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-04-01T05:04:57.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:04:57.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:04:57.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-04-01T05:04:57.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:04:57.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-04-01T05:04:57.418 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509449 2024-04-01T05:04:57.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-01T05:04:57.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:04:57.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509449 2024-04-01T05:04:57.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:04:57.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:04:57.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:04:58.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:04:58.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:04:58.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:04:58.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:04:58.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:04:58.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:04:58.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:04:58.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:04:58.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:04:58.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:04:58.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:04:58.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:04:58.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:04:58.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-01T05:04:58.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:04:58.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-01T05:04:58.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:04:58.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:04:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:04:59.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:04:59.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:04:59.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:04:59.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:04:59.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:04:59.424 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:04:59.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:04:59.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:04:59.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:04:59.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:04:59.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:04:59.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:05:00.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:05:00.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:05:00.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:00.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-04-01T05:05:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:00.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:00.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:00.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:00.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:00.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-04-01T05:05:01.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:05:01.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:05:02.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:05:02.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:05:02.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:05:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:05:02.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:05:02.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:05:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:05:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:05:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:05:02.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:05:02.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-04-01T05:05:02.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:05:02.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:05:02.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:05:02.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:05:02.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:05:02.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-04-01T05:05:03.946 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:ff7b1f36:::obj1:head# 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:05:04.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:04.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:05:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:05:04.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:04.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:05:04.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:05:04.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:05:04.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:05:04.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:05:04.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:04.495+0000 7f6e4a814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:04.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:04.497+0000 7f6e4a814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:04.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:04.498+0000 7f6e4a814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:04.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:05:04.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:05:04.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:05:04.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:05:04.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:05:04.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:04.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:05:04.760 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:05:04.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:04.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:05.013 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:05.013+0000 7f6e4a814640 -1 Falling back to public interface 2024-04-01T05:05:05.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:06.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:06.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:06.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:05:06.094 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:06.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:06.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:07.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:07.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:07.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:05:07.389 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:05:07.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:07.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:07.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:07.543+0000 7f6e4a814640 -1 osd.0 21 log_to_monitors true 2024-04-01T05:05:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:08.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:08.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:08.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:05:08.742 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:05:08.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:08.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/323460377,v1:127.0.0.1:6803/323460377] [v2:127.0.0.1:6804/323460377,v1:127.0.0.1:6805/323460377] exists,up 36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:05:09.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:05:09.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:05:09.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:05:09.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:05:09.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:05:09.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:05:09.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:05:09.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:05:09.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:05:09.448 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:09.448 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:05:09.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:05:09.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:09.448 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:05:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-04-01T05:05:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-04-01T05:05:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-04-01T05:05:09.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:09.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:05:09.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-04-01T05:05:09.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-04-01T05:05:09.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672974' 2024-04-01T05:05:09.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:09.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:05:09.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-04-01T05:05:09.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-04-01T05:05:09.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672974 2-64424509453' 2024-04-01T05:05:09.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:09.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-04-01T05:05:09.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:09.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:05:09.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-04-01T05:05:09.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-04-01T05:05:09.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-04-01T05:05:09.766 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182402 2024-04-01T05:05:09.766 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:10.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-04-01T05:05:10.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:05:11.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:05:11.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:11.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-04-01T05:05:11.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:11.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-04-01T05:05:11.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:11.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:05:11.346 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-04-01T05:05:11.346 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:11.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-04-01T05:05:11.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-04-01T05:05:11.348 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672974 2024-04-01T05:05:11.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:05:11.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-04-01T05:05:11.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:11.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-04-01T05:05:11.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:11.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:05:11.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-04-01T05:05:11.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:11.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-04-01T05:05:11.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-01T05:05:11.635 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509453 2024-04-01T05:05:11.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:05:11.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-04-01T05:05:11.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:05:11.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:11.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:12.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:05:12.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:05:12.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:05:12.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:05:12.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:05:12.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:05:12.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:05:12.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:05:12.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:05:12.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:05:12.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:12.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:05:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:05:13.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:05:13.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:05:13.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:05:13.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-04-01T05:05:13.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:05:13.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:05:13.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:05:13.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:05:13.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:05:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:05:13.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:05:13.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-04-01T05:05:13.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:05:13.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:05:13.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:05:13.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:05:13.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:05:13.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-04-01T05:05:14.978 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:104778fc:::obj2:head# 2024-04-01T05:05:15.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:05:15.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:05:15.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:05:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:05:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:05:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:05:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:05:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:05:15.499 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:05:15.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:15.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:05:15.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:05:15.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:05:15.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:05:15.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:05:15.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:15.527+0000 7ff0b240d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:15.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:15.529+0000 7ff0b240d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:15.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:15.531+0000 7ff0b240d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:16.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:16.045+0000 7ff0b240d640 -1 Falling back to public interface 2024-04-01T05:05:16.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:17.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:17.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:17.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:05:17.079 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:17.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:17.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:17.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:05:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:05:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:18.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:18.466+0000 7ff0b240d640 -1 osd.0 26 log_to_monitors true 2024-04-01T05:05:18.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:19.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:19.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:19.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:05:19.674 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:05:19.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:19.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:19.965 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/652054551,v1:127.0.0.1:6803/652054551] [v2:127.0.0.1:6804/652054551,v1:127.0.0.1:6805/652054551] exists,up 36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:05:19.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:05:19.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:05:20.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:05:20.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:05:20.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:05:20.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:05:20.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:05:20.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:05:20.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:05:20.330 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:20.330 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:05:20.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:05:20.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:20.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:05:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-04-01T05:05:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-04-01T05:05:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-04-01T05:05:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:20.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:05:20.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-04-01T05:05:20.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-04-01T05:05:20.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672977' 2024-04-01T05:05:20.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:20.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:05:20.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-04-01T05:05:20.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-04-01T05:05:20.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672977 2-64424509456' 2024-04-01T05:05:20.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:20.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-04-01T05:05:20.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:20.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:05:20.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-04-01T05:05:20.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:20.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-04-01T05:05:20.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-04-01T05:05:20.648 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 128849018882 2024-04-01T05:05:20.648 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:20.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-04-01T05:05:20.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:05:21.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:05:21.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-04-01T05:05:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:22.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-04-01T05:05:22.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:05:22.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-04-01T05:05:22.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-04-01T05:05:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-01T05:05:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672977 2024-04-01T05:05:22.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:05:22.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-01T05:05:22.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:22.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-04-01T05:05:22.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:22.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:05:22.512 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-04-01T05:05:22.512 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:22.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-04-01T05:05:22.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-01T05:05:22.513 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509456 2024-04-01T05:05:22.513 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:05:22.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-04-01T05:05:22.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:05:22.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:22.797 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:23.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:05:23.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:05:23.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:05:23.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:05:23.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:23.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:05:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:05:23.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:05:23.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-04-01T05:05:23.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:05:23.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:05:23.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:05:23.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:05:24.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:05:24.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:05:24.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:05:24.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-04-01T05:05:24.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:05:24.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:05:24.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:05:24.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:05:24.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:05:24.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:05:26.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:26.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:26.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:05:26.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:05:26.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:05:26.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:05:26.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:05:26.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:26.215+0000 7fddcc80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:26.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:26.217+0000 7fddcc80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:26.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:26.218+0000 7fddcc80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:26.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:05:26.744 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:26.744+0000 7fddcc80d640 -1 Falling back to public interface 2024-04-01T05:05:26.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:05:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:27.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:05:28.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:29.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:29.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:29.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:05:29.058 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:05:29.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:29.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:05:29.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:29.172+0000 7fddcc80d640 -1 osd.1 31 log_to_monitors true 2024-04-01T05:05:29.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:30.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:30.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:30.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:05:30.383 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:05:30.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:30.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:05:30.673 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/2177528534,v1:127.0.0.1:6811/2177528534] [v2:127.0.0.1:6812/2177528534,v1:127.0.0.1:6813/2177528534] exists,up 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d 2024-04-01T05:05:30.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:05:30.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:05:30.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:05:30.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:05:30.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:05:30.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:05:30.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:05:30.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:05:31.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:05:31.041 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:31.041 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:05:31.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:05:31.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:31.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:05:31.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-04-01T05:05:31.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-04-01T05:05:31.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-04-01T05:05:31.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:31.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:05:31.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-04-01T05:05:31.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-04-01T05:05:31.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-04-01T05:05:31.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:31.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:05:31.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-04-01T05:05:31.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-04-01T05:05:31.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509460' 2024-04-01T05:05:31.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:31.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-04-01T05:05:31.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:31.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:05:31.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-04-01T05:05:31.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:31.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-04-01T05:05:31.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-04-01T05:05:31.359 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 128849018885 2024-04-01T05:05:31.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:31.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-04-01T05:05:31.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:05:32.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:05:32.643 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:32.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-04-01T05:05:32.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:32.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-04-01T05:05:32.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:32.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:05:32.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-04-01T05:05:32.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:32.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-04-01T05:05:32.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-04-01T05:05:32.941 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 150323855362 2024-04-01T05:05:32.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:05:33.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-04-01T05:05:33.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:33.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-04-01T05:05:33.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:33.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:05:33.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-04-01T05:05:33.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:33.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-04-01T05:05:33.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-01T05:05:33.230 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509460 2024-04-01T05:05:33.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:05:33.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-04-01T05:05:33.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:05:33.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:33.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:33.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:05:33.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:05:33.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:05:33.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:05:33.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:05:33.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:05:33.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:05:33.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:05:34.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:05:34.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:05:34.193 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:34.193 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-01T05:05:34.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-04-01T05:05:34.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:05:34.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-04-01T05:05:34.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-04-01T05:05:34.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:05:34.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T05:05:34.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T05:05:34.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:05:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-04-01T05:05:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-04-01T05:05:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:05:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:05:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:05:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:05:35.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:35.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-04-01T05:05:35.498 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T05:05:35.498 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": true, 2024-04-01T05:05:35.498 INFO:tasks.workunit.client.0.smithi067.stdout: "must": false, 2024-04-01T05:05:35.498 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "2024-03-18T05:03:55.498190+0000" 2024-04-01T05:05:35.498 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:05:35.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:35.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T05:05:35.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:35.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:05:35.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:05:35.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:05:35.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:05:35.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:05:35.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:05:35.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:05:35.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:05:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:04:51.848855+0000 '>' 2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:35.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:05:36.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:05:36.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:05:36.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:05:36.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:05:36.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:05:36.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:05:36.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:05:37.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:04:51.848855+0000 '>' 2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:37.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:05:38.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:05:38.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:05:38.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:05:38.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:05:38.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:05:38.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:05:38.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:05:38.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:04:51.848855+0000 '>' 2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:38.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:05:39.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:05:39.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:05:39.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:05:39.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:05:39.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:05:39.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:05:39.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:05:39.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:04:51.848855+0000 '>' 2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:39.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:05:40.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:05:40.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:05:40.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:05:40.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:05:40.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:05:40.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:05:40.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:05:37.418108+0000 '>' 2024-04-01T05:04:51.848855+0000 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:05:40.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:05:40.959 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:05:40.959 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:05:40.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:05:40.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:05:40.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:05:41.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:05:41.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:05:41.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:05:41.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:05:41.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:05:41.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:05:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:05:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:05:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:05:41.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:41.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:05:41.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018888 2024-04-01T05:05:41.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018888 2024-04-01T05:05:41.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888' 2024-04-01T05:05:41.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:41.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:05:41.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-04-01T05:05:41.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-04-01T05:05:41.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-04-01T05:05:41.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:41.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:05:41.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-04-01T05:05:41.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-04-01T05:05:41.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509463' 2024-04-01T05:05:41.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:41.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018888 2024-04-01T05:05:41.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:41.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:05:41.638 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018888 2024-04-01T05:05:41.638 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:41.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018888 2024-04-01T05:05:41.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-04-01T05:05:41.639 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 128849018888 2024-04-01T05:05:41.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:41.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018888 2024-04-01T05:05:41.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:05:42.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:05:42.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018888 2024-04-01T05:05:43.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:43.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-04-01T05:05:43.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:43.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:05:43.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-04-01T05:05:43.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-04-01T05:05:43.242 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 150323855365 2024-04-01T05:05:43.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-04-01T05:05:43.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:05:43.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-04-01T05:05:43.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:43.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-04-01T05:05:43.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:43.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:05:43.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-04-01T05:05:43.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:43.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-04-01T05:05:43.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-04-01T05:05:43.527 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509463 2024-04-01T05:05:43.527 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:05:43.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-04-01T05:05:43.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:05:43.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:43.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:05:44.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:05:44.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:05:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:05:44.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:05:44.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:44.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:44.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:05:44.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:05:44.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:05:44.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-04-01T05:05:44.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:05:44.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:05:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:05:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:45.099 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:05:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:05:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:45.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:05:45.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-04-01T05:05:45.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-04-01T05:05:45.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-04-01T05:05:45.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:45.205 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:05:45.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-04-01T05:05:45.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-04-01T05:05:45.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-04-01T05:05:45.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:45.309 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:05:45.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-04-01T05:05:45.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-04-01T05:05:45.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509464' 2024-04-01T05:05:45.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:45.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-04-01T05:05:45.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:45.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:05:45.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-04-01T05:05:45.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:45.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-04-01T05:05:45.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-04-01T05:05:45.416 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 128849018890 2024-04-01T05:05:45.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:45.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-04-01T05:05:45.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:05:46.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:05:46.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:46.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-04-01T05:05:46.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:46.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-04-01T05:05:46.989 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:46.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:05:46.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-04-01T05:05:46.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:46.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-04-01T05:05:46.991 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 150323855367 2024-04-01T05:05:46.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-04-01T05:05:46.992 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:05:47.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-04-01T05:05:47.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:47.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-04-01T05:05:47.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:47.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:05:47.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-04-01T05:05:47.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:47.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-04-01T05:05:47.304 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509464 2024-04-01T05:05:47.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-04-01T05:05:47.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:05:47.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509464 2024-04-01T05:05:47.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-04-01T05:05:47.590 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:05:47.593 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:05:47.593 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-04-01T05:05:47.593 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-04-01T05:05:47.593 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:05:47.594 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:05:47.594 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-04-01T05:05:47.594 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:05:47.594 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-04-01T05:05:47.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-04-01T05:05:47.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-04-01T05:05:47.860 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:05:47.893 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-04-01T05:05:37.418187+0000 21'10 36:96 [1,0] 1 [1,0] 1 21'10 2024-04-01T05:05:37.418108+0000 21'10 2024-04-01T05:05:37.418108+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:05:37.418108+0000 10 0 2024-04-01T05:05:47.893 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:05:47.893 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:05:47.894 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:05:47.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-04-01T05:05:47.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-04-01T05:05:48.137 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:05:48.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T05:05:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:05:48.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:05:48.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-04-01T05:05:48.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:05:48.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-04-01T05:05:48.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:05:48.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:05:49.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:05:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-04-01T05:05:50.240 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:05:50.241 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:05:50.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:50.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:05:50.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:05:50.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:05:50.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:05:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:05:50.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:05:50.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:05:50.511 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:05:50.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:05:50.512 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:05:50.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:05:50.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:05:50.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:05:50.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:05:50.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:50.540+0000 7fe19bc22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:50.543 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:50.543+0000 7fe19bc22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:50.544 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:50.544+0000 7fe19bc22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:50.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:51.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:51.068+0000 7fe19bc22640 -1 Falling back to public interface 2024-04-01T05:05:51.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:52.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:52.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:52.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:05:52.092 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:52.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:52.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:52.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:53.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:53.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:53.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:05:53.385 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:05:53.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:53.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:05:53.493+0000 7fe19bc22640 -1 osd.0 36 log_to_monitors true 2024-04-01T05:05:53.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:05:54.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:05:54.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:05:54.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:05:54.714 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:05:54.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:05:54.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:05:55.007 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/1579695703,v1:127.0.0.1:6803/1579695703] [v2:127.0.0.1:6804/1579695703,v1:127.0.0.1:6805/1579695703] exists,up 36d59fbc-bfc7-4330-895f-6863e02fe98b 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:05:55.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:05:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:05:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:05:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:05:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:05:55.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:05:55.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:05:55.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:05:55.375 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:05:55.375 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:05:55.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:05:55.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:55.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:05:55.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-04-01T05:05:55.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-04-01T05:05:55.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-04-01T05:05:55.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:55.482 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:05:55.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-04-01T05:05:55.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-04-01T05:05:55.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-04-01T05:05:55.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:05:55.612 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:05:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-04-01T05:05:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-04-01T05:05:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509467' 2024-04-01T05:05:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:55.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-04-01T05:05:55.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:05:55.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-04-01T05:05:55.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:55.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-04-01T05:05:55.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-04-01T05:05:55.721 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 171798691842 2024-04-01T05:05:55.721 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-04-01T05:05:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:05:57.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:05:57.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:05:57.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-04-01T05:05:57.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:57.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-04-01T05:05:57.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:57.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:05:57.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-04-01T05:05:57.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:57.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-04-01T05:05:57.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-04-01T05:05:57.305 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 150323855370 2024-04-01T05:05:57.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:05:57.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-04-01T05:05:57.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:05:57.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-04-01T05:05:57.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:05:57.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:05:57.590 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-04-01T05:05:57.590 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:05:57.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-04-01T05:05:57.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-04-01T05:05:57.591 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509467 2024-04-01T05:05:57.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:05:57.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509467 2024-04-01T05:05:57.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:05:57.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:57.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:05:58.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:05:58.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:05:58.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:05:58.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:05:58.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:05:58.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:05:58.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:05:58.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:05:58.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-04-01T05:05:58.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:05:58.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-04-01T05:05:58.899 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:05:58.899 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:05:58.899 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-04-01T05:05:58.899 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-04-01T05:05:58.899 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:05:58.899 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:05:58.900 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-04-01T05:05:58.900 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:05:58.900 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-04-01T05:05:58.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-04-01T05:05:58.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:05:58.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:05:58.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:05:58.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:05:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-04-01T05:05:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:05:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:05:59.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:05:59.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-04-01T05:05:59.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:05:59.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:05:59.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:05:59.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:05:59.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:05:59.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-04-01T05:06:00.524 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-04-01T05:06:00.524 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:104778fc:::obj2:head# 2024-04-01T05:06:01.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:06:01.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:06:01.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:06:01.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:06:01.050 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:06:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:06:01.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:06:01.052 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:06:01.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:06:01.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:06:01.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:06:01.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:06:01.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:06:01.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:06:01.082 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:01.080+0000 7f8354c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:01.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:01.091+0000 7f8354c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:01.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:01.093+0000 7f8354c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:01.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:06:01.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:01.608+0000 7f8354c13640 -1 Falling back to public interface 2024-04-01T05:06:01.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:06:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:06:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:06:02.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:03.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:03.903+0000 7f8354c13640 -1 osd.1 41 log_to_monitors true 2024-04-01T05:06:03.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:03.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:03.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:06:03.922 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:06:03.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:03.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:06:04.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:05.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:06:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:06:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:05.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3469751774,v1:127.0.0.1:6811/3469751774] [v2:127.0.0.1:6812/3469751774,v1:127.0.0.1:6813/3469751774] exists,up 38ba8b3a-cc14-4bb3-b0b2-b2a3d1f8962d 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:06:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:06:05.511 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:06:05.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:06:05.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:06:05.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:06:05.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:06:05.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:06:05.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:06:05.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:06:05.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:06:05.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:06:05.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:06:05.893 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:06:05.893 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:06:05.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:06:05.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:06:05.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:06:05.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-04-01T05:06:05.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-04-01T05:06:05.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-04-01T05:06:05.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:06:05.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:06:06.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-04-01T05:06:06.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-04-01T05:06:06.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-04-01T05:06:06.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:06:06.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:06:06.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-04-01T05:06:06.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-04-01T05:06:06.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509471' 2024-04-01T05:06:06.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:06:06.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-04-01T05:06:06.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:06:06.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:06:06.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-04-01T05:06:06.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:06:06.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-04-01T05:06:06.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-04-01T05:06:06.214 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 171798691845 2024-04-01T05:06:06.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:06:06.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-04-01T05:06:06.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:06:06.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-04-01T05:06:06.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:06:06.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:06:06.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-04-01T05:06:06.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:06:06.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-04-01T05:06:06.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-04-01T05:06:06.499 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 193273528322 2024-04-01T05:06:06.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:06:06.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 193273528322 2024-04-01T05:06:06.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:06:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:06:07.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:06:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 193273528322 2024-04-01T05:06:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:06:09.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:06:09.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:06:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-04-01T05:06:09.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:06:09.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-04-01T05:06:09.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:06:09.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:06:09.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-04-01T05:06:09.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:06:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-04-01T05:06:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-01T05:06:09.380 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509471 2024-04-01T05:06:09.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:06:09.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-04-01T05:06:09.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:06:09.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:06:09.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:06:10.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:06:10.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:06:10.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:06:10.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:06:10.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:06:10.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:06:10.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:06:10.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:06:10.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:06:10.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:06:10.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:06:10.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:06:10.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:06:11.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:11.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-04-01T05:06:11.295 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 1.0 on osd.1 to repair 2024-04-01T05:06:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T05:06:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:06:11.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:06:11.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:05:37.418108+0000 '>' 2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:11.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:06:12.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:06:12.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:06:12.590 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:06:12.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:06:12.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:06:12.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:06:12.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:06:12.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:05:37.418108+0000 '>' 2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:12.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:06:13.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:06:13.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:06:13.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:06:13.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:06:13.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:06:13.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:06:13.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:06:14.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:05:37.418108+0000 '>' 2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:14.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:06:15.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:06:15.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:06:15.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:06:15.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:06:15.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:06:15.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:06:15.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:06:15.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:05:37.418108+0000 '>' 2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:15.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:06:16.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:06:16.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:06:16.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:06:16.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:06:16.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:06:16.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:06:16.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:06:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:06:11.943139+0000 '>' 2024-04-01T05:05:37.418108+0000 2024-04-01T05:06:16.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:06:16.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-04-01T05:06:18.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-04-01T05:06:18.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:06:18.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:06:19.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:06:19.047 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:06:19.047 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:06:19.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:06:19.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:06:19.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:06:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-04-01T05:06:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-04-01T05:06:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849' 2024-04-01T05:06:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:06:19.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:06:19.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528326 2024-04-01T05:06:19.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528326 2024-04-01T05:06:19.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528326' 2024-04-01T05:06:19.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:06:19.279 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:06:19.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-04-01T05:06:19.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-04-01T05:06:19.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528326 2-64424509474' 2024-04-01T05:06:19.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:06:19.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691849 2024-04-01T05:06:19.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:06:19.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:06:19.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691849 2024-04-01T05:06:19.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:06:19.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-04-01T05:06:19.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-04-01T05:06:19.387 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 171798691849 2024-04-01T05:06:19.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:06:19.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691849 2024-04-01T05:06:19.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:06:20.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:06:20.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:06:20.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-04-01T05:06:20.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:06:20.965 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528326 2024-04-01T05:06:20.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:06:20.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:06:20.967 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528326 2024-04-01T05:06:20.967 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:06:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528326 2024-04-01T05:06:20.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528326' 2024-04-01T05:06:20.968 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 193273528326 2024-04-01T05:06:20.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:06:21.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528326 -lt 193273528326 2024-04-01T05:06:21.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:06:21.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-04-01T05:06:21.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:06:21.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:06:21.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-04-01T05:06:21.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:06:21.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-04-01T05:06:21.255 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509474 2024-04-01T05:06:21.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-04-01T05:06:21.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:06:21.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-04-01T05:06:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-04-01T05:06:21.811 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:06:21.811 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-04-01T05:06:11.943230+0000 21'10 46:134 [1,0] 1 [1,0] 1 21'10 2024-04-01T05:06:11.943139+0000 21'10 2024-04-01T05:06:11.943139+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:06:11.943139+0000 9 0 2024-04-01T05:06:21.811 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:06:21.811 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:06:21.811 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:06:21.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-04-01T05:06:21.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-04-01T05:06:22.087 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:06:22.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-04-01T05:06:22.101 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:06:22.101 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:06:22.101 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-04-01T05:06:22.101 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.149+0000 7fddb013c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:05:37.417+0000 7fddb013c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:06:11.942+0000 7f83384c8640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:06:11.942+0000 7f83384c8640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:06:11.942+0000 7f83384c8640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:06:11.942+0000 7f83384c8640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-04-01T05:06:22.102 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:06:11.942+0000 7f83384c8640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:06:22.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:06:22.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:06:22.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:06:22.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:06:22.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:06:22.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:06:22.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:06:22.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:06:22.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:06:22.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:06:22.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:06:22.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:06:22.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:06:22.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:06:22.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:06:22.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:06:22.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:06:22.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:22.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:22.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:06:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:06:22.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:06:22.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:06:22.235 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:06:22.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:06:22.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:06:22.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:06:22.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:06:22.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:06:22.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:06:22.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:06:22.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:06:22.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:06:22.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:06:22.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:06:22.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:22.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:22.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:06:22.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:06:22.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:06:22.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:06:22.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:06:22.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:22.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:22.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:06:22.245 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:06:22.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:06:22.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:06:22.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:22.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:06:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:22.824 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:06:22.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:06:22.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:06:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:06:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:06:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:06:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:06:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:06:22.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:06:22.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-04-01T05:06:22.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:06:22.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:06:22.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:06:22.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:06:22.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:06:22.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:06:23.798 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:06:23.798 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:06:23.798 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:06:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:06:23.798 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:23.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:23.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:06:23.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:06:23.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:06:23.830 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-04-01T05:06:23.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-04-01T05:06:23.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:06:23.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:06:23.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:06:23.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:06:23.831 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:06:23.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:06:23.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:06:23.833 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:06:23.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f106366-a6fa-4b81-8beb-e7d06872eec5 2024-04-01T05:06:23.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1f106366-a6fa-4b81-8beb-e7d06872eec5' 2024-04-01T05:06:23.833 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 1f106366-a6fa-4b81-8beb-e7d06872eec5 2024-04-01T05:06:23.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:06:23.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPQApmflmKMhAApJOopCmKqKrSu8LF2elMeg== 2024-04-01T05:06:23.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPQApmflmKMhAApJOopCmKqKrSu8LF2elMeg=="}' 2024-04-01T05:06:23.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f106366-a6fa-4b81-8beb-e7d06872eec5 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:06:24.088 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:06:24.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:06:24.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDPQApmflmKMhAApJOopCmKqKrSu8LF2elMeg== --osd-uuid 1f106366-a6fa-4b81-8beb-e7d06872eec5 2024-04-01T05:06:24.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.129+0000 7ff8e99ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:24.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.138+0000 7ff8e99ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:24.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.139+0000 7ff8e99ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:24.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.139+0000 7ff8e99ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:06:24.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.139+0000 7ff8e99ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:06:24.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.139+0000 7ff8e99ac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:06:24.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:24.230+0000 7ff8e99ac640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:06:29.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:06:29.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:06:29.704 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:06:29.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:06:29.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:06:30.073 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:06:30.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:06:30.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:06:30.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:06:30.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:06:30.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:06:30.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:30.100+0000 7f4136011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:30.102 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:30.102+0000 7f4136011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:30.103+0000 7f4136011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:30.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:06:30.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:06:30.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:06:30.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:06:30.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:06:30.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:30.374 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:06:30.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:06:30.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:30.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:30.631 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:30.631+0000 7f4136011640 -1 Falling back to public interface 2024-04-01T05:06:30.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:31.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:06:31.657 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:06:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:31.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:31.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:32.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:32.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:32.951 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:06:32.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:06:32.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:32.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:32.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:32.960+0000 7f4136011640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:06:33.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:34.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:34.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:34.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:06:34.379 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:06:34.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:34.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:34.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:35.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:35.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:35.672 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:06:35.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:06:35.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:35.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:36.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:36.968 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:06:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:06:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:36.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:37.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:38.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:38.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:38.263 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:06:38.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:06:38.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:38.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:38.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:39.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:39.670 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:06:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:06:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:39.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:39.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:40.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:40.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:40.965 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:06:40.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:06:40.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:40.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:41.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:42.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:42.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:42.258 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:06:42.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:06:42.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:42.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:42.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:43.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:43.551 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:06:43.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:43.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:06:43.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:43.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:44.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:45.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:45.217 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:06:45.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:45.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:06:45.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:45.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:45.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:46.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:46.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:46.795 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:06:46.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:06:46.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:46.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:47.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:48.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:48.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:48.372 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:06:48.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:06:48.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:48.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:48.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:49.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:49.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:49.966 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:06:49.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:06:49.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:49.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:50.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:51.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:51.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:51.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:06:51.878 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:06:51.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:51.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:52.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:53.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:53.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:53.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:06:53.590 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:06:53.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:53.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:54.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:06:55.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:06:55.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:06:55.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:06:55.275 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:06:55.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:06:55.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:06:55.566 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1064833165,v1:127.0.0.1:6803/1064833165] [v2:127.0.0.1:6804/1064833165,v1:127.0.0.1:6805/1064833165] exists,up 1f106366-a6fa-4b81-8beb-e7d06872eec5 2024-04-01T05:06:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:06:55.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:06:55.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:06:55.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9313908f-bae5-412a-abdf-941d76ee6b73 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9313908f-bae5-412a-abdf-941d76ee6b73' 2024-04-01T05:06:55.573 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 9313908f-bae5-412a-abdf-941d76ee6b73 2024-04-01T05:06:55.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:06:55.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvQApm/4kPIxAAiE0bCXYt70vGg1sK++gbiQ== 2024-04-01T05:06:55.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvQApm/4kPIxAAiE0bCXYt70vGg1sK++gbiQ=="}' 2024-04-01T05:06:55.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9313908f-bae5-412a-abdf-941d76ee6b73 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:06:55.960 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:06:55.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:06:55.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDvQApm/4kPIxAAiE0bCXYt70vGg1sK++gbiQ== --osd-uuid 9313908f-bae5-412a-abdf-941d76ee6b73 2024-04-01T05:06:56.001 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.001+0000 7fe9c8010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:56.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.044+0000 7fe9c8010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:56.045 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.045+0000 7fe9c8010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:06:56.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.045+0000 7fe9c8010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:06:56.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.045+0000 7fe9c8010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:06:56.046 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.045+0000 7fe9c8010640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:06:56.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:06:56.110+0000 7fe9c8010640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:07:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:07:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:07:02.116 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:07:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:07:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:07:02.486 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:07:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:07:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:07:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:07:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:07:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:07:02.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:02.514+0000 7f6ac54ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:02.516 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:02.516+0000 7f6ac54ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:02.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:02.517+0000 7f6ac54ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:02.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:07:02.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:07:02.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:07:02.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:07:02.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:07:02.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:02.771 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:07:02.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:07:02.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:02.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:03.039 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:03.039+0000 7f6ac54ce640 -1 Falling back to public interface 2024-04-01T05:07:03.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:04.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:04.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:04.073 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:07:04.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:07:04.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:04.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:04.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:05.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:05.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:05.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:07:05.368 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:07:05.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:05.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:05.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:05.434+0000 7f6ac54ce640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:07:05.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:06.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:06.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:06.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:07:06.663 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:07:06.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:06.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:06.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:07.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:07.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:07.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:07:07.956 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:07:07.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:07.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:08.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:09.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:09.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:09.262 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:07:09.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:07:09.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:09.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:09.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:10.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:10.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:10.639 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:07:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:07:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:10.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:10.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:11.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:11.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:11.936 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:07:11.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:07:11.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:11.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:12.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:13.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:13.313 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:07:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:07:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:13.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:13.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:14.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:14.681 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:07:14.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:14.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:07:14.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:14.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:14.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:15.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:15.977 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:07:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:07:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:15.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:16.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:17.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:17.287 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:07:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:07:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:17.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:17.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:18.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:18.881 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:07:18.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:18.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:07:18.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:18.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:19.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:20.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:20.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:20.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:07:20.501 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:07:20.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:20.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:22.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:07:22.322 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:07:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:22.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:23.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:23.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:23.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:07:23.806 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:07:23.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:23.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:24.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:25.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:25.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:25.229 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:07:25.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:07:25.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:25.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:26.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:27.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:27.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:27.070 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:07:27.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:07:27.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:27.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:27.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:28.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:28.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:28.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:07:28.373 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:07:28.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:28.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:07:28.717 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2764534261,v1:127.0.0.1:6811/2764534261] [v2:127.0.0.1:6812/2764534261,v1:127.0.0.1:6813/2764534261] exists,up 9313908f-bae5-412a-abdf-941d76ee6b73 2024-04-01T05:07:28.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:07:28.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:07:28.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:07:28.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:07:28.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:07:28.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cf16dc2-f622-401b-b290-676f14f154fc 2024-04-01T05:07:28.724 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 5cf16dc2-f622-401b-b290-676f14f154fc 2024-04-01T05:07:28.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5cf16dc2-f622-401b-b290-676f14f154fc' 2024-04-01T05:07:28.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:07:28.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQQQpmQLbqKxAAKqzxkChPSTlVe51ipqq3Yw== 2024-04-01T05:07:28.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQQQpmQLbqKxAAKqzxkChPSTlVe51ipqq3Yw=="}' 2024-04-01T05:07:28.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cf16dc2-f622-401b-b290-676f14f154fc -i td/osd-scrub-repair/2/new.json 2024-04-01T05:07:29.099 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:07:29.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:07:29.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAQQQpmQLbqKxAAKqzxkChPSTlVe51ipqq3Yw== --osd-uuid 5cf16dc2-f622-401b-b290-676f14f154fc 2024-04-01T05:07:29.141 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.139+0000 7f6352c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:29.175 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.174+0000 7f6352c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:29.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.175+0000 7f6352c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:29.176 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.175+0000 7f6352c0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:07:29.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.175+0000 7f6352c0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:07:29.177 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.175+0000 7f6352c0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:07:29.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:29.240+0000 7f6352c0e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:07:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:07:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:07:35.199 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:07:35.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:07:35.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:07:35.568 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:07:35.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:07:35.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:07:35.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:07:35.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:07:35.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:07:35.597 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:35.597+0000 7f56f9eb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:35.599 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:35.599+0000 7f56f9eb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:35.600 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:35.600+0000 7f56f9eb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:07:35.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:07:35.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:07:35.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:07:35.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:07:35.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:07:35.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:35.854 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:07:35.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:07:35.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:35.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:36.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:36.121+0000 7f56f9eb1640 -1 Falling back to public interface 2024-04-01T05:07:36.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:37.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:37.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:37.136 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:07:37.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:07:37.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:37.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:37.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:38.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:38.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:38.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:07:38.435 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:07:38.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:38.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:38.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:07:38.490+0000 7f56f9eb1640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:07:38.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:39.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:39.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:39.731 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:07:39.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:07:39.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:39.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:40.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:41.064 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:07:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:07:41.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:41.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:41.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:42.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:42.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:42.358 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:07:42.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:07:42.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:42.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:42.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:43.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:43.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:43.653 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:07:43.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:07:43.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:43.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:43.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:44.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:44.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:44.950 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:07:44.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:07:44.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:44.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:45.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:46.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:46.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:46.454 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:07:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:07:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:46.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:47.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:47.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:47.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:07:47.750 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:07:47.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:47.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:07:49.047 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:07:49.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:49.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:49.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:50.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:50.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:50.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:07:50.409 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:07:50.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:50.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:51.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:52.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:52.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:52.140 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:07:52.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:07:52.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:52.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:52.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:53.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:53.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:53.935 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:07:53.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:07:53.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:53.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:54.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:55.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:55.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:55.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:07:55.355 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:07:55.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:55.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:55.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:56.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:56.965 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:07:56.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:56.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:07:56.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:56.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:58.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:58.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:58.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:07:58.309 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:07:58.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:58.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:07:58.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:07:59.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:07:59.877 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:07:59.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:07:59.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:07:59.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:07:59.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:08:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:01.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:01.371 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:08:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:08:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:01.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:08:01.666 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1767936007,v1:127.0.0.1:6819/1767936007] [v2:127.0.0.1:6820/1767936007,v1:127.0.0.1:6821/1767936007] exists,up 5cf16dc2-f622-401b-b290-676f14f154fc 2024-04-01T05:08:01.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:08:01.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:08:01.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:08:01.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-04-01T05:08:01.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-04-01T05:08:02.091 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T05:08:02.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:08:03.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-04-01T05:08:03.488 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 2 2024-04-01T05:08:03.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-04-01T05:08:03.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:08:03.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:08:03.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:08:03.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:08:03.499 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:08:03.500 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:08:03.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:08:03.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:08:03.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:08:03.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:08:03.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:08:03.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:08:03.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:08:03.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:08:03.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:08:03.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:08:03.876 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:03.876 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:08:03.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:08:03.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:03.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:08:03.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-04-01T05:08:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-04-01T05:08:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-04-01T05:08:03.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:03.982 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:08:04.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-04-01T05:08:04.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-04-01T05:08:04.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977' 2024-04-01T05:08:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:04.161 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:08:04.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-04-01T05:08:04.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-04-01T05:08:04.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509446' 2024-04-01T05:08:04.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:04.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-04-01T05:08:04.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:04.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:08:04.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-04-01T05:08:04.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:04.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-04-01T05:08:04.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-04-01T05:08:04.269 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836501 2024-04-01T05:08:04.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:04.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-04-01T05:08:04.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:08:05.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:08:05.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:05.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-04-01T05:08:05.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:08:06.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:08:06.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:07.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-04-01T05:08:07.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:07.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-04-01T05:08:07.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:07.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:08:07.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-04-01T05:08:07.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:07.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-04-01T05:08:07.197 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672977 2024-04-01T05:08:07.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-01T05:08:07.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:08:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-01T05:08:07.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:07.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-04-01T05:08:07.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:07.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:08:07.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-04-01T05:08:07.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:07.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-04-01T05:08:07.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-01T05:08:07.480 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509446 2024-04-01T05:08:07.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:08:07.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-04-01T05:08:07.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:08:07.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:07.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:08:08.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:08:08.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:08:08.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:08:08.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:08.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:08.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:08.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:08.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:08.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:09.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:09.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:10.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-04-01T05:08:10.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:08:10.130 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:08:10.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:08:10.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:08:10.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:08:10.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:08:10.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:08:10.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:08:11.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:08:11.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-04-01T05:08:11.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:08:11.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:08:11.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:08:11.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:08:11.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:08:11.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-04-01T05:08:12.272 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:ff7b1f36:::obj1:head# 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:08:12.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:08:12.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:08:12.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:12.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:08:12.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:08:12.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:08:12.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:08:12.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:08:12.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:12.821+0000 7ff414815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:12.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:12.823+0000 7ff414815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:12.826 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:12.824+0000 7ff414815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:08:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:08:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:08:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:08:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:08:13.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:13.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:08:13.090 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:08:13.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:13.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:13.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:13.339+0000 7ff414815640 -1 Falling back to public interface 2024-04-01T05:08:13.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:08:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:14.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:14.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:15.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:15.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:15.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:08:15.667 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:08:15.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:15.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:15.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:15.703+0000 7ff414815640 -1 osd.0 21 log_to_monitors true 2024-04-01T05:08:16.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:17.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:17.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:17.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:08:17.017 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:08:17.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:17.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:17.338 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/107498714,v1:127.0.0.1:6803/107498714] [v2:127.0.0.1:6804/107498714,v1:127.0.0.1:6805/107498714] exists,up 1f106366-a6fa-4b81-8beb-e7d06872eec5 2024-04-01T05:08:17.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:08:17.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:08:17.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:08:17.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:08:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:08:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:08:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:08:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:08:17.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:08:17.422 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:08:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:08:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:08:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:08:17.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:17.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:08:17.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-04-01T05:08:17.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-04-01T05:08:17.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-04-01T05:08:17.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:17.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:08:17.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-04-01T05:08:17.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-04-01T05:08:17.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-04-01T05:08:17.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:17.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509450' 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:18.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:08:18.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-04-01T05:08:18.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:18.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-04-01T05:08:18.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-04-01T05:08:18.071 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182402 2024-04-01T05:08:18.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:18.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-04-01T05:08:18.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:18.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-04-01T05:08:18.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:18.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:08:18.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-04-01T05:08:18.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:18.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-04-01T05:08:18.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-04-01T05:08:18.388 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672980 2024-04-01T05:08:18.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:08:18.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-04-01T05:08:18.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:18.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-04-01T05:08:18.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:18.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:08:18.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-04-01T05:08:18.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-04-01T05:08:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-01T05:08:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509450 2024-04-01T05:08:18.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:08:19.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509450 2024-04-01T05:08:19.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:08:20.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:08:20.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:08:20.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-01T05:08:20.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:08:20.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:20.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:20.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:08:20.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:08:20.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:08:20.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:08:20.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:08:20.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:08:20.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:08:20.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:08:20.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:08:20.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:08:20.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:20.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:08:21.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:08:21.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-04-01T05:08:21.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:08:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:08:21.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:08:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:08:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-04-01T05:08:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:08:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:08:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:08:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:08:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:08:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-04-01T05:08:23.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:23.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:08:23.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:08:23.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:08:23.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:08:23.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:08:23.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:08:23.778 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:08:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:23.779 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:08:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:08:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:08:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:08:23.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:08:23.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:23.807+0000 7f1480ab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:23.809 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:23.809+0000 7f1480ab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:23.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:23.810+0000 7f1480ab5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:24.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:24.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:24.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:24.327+0000 7f1480ab5640 -1 Falling back to public interface 2024-04-01T05:08:24.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:25.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:25.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:25.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:08:25.354 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:25.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:25.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:25.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:26.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:26.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:26.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:08:26.648 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:08:26.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:26.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:26.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:26.666+0000 7f1480ab5640 -1 osd.1 26 log_to_monitors true 2024-04-01T05:08:26.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:27.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:27.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:27.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:08:27.980 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:08:27.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:27.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:28.275 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/923788702,v1:127.0.0.1:6811/923788702] [v2:127.0.0.1:6812/923788702,v1:127.0.0.1:6813/923788702] exists,up 9313908f-bae5-412a-abdf-941d76ee6b73 2024-04-01T05:08:28.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:08:28.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:08:28.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:08:28.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:08:28.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:08:28.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:08:28.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:08:28.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:08:28.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:08:28.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:08:28.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:08:28.639 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:28.639 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:08:28.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:08:28.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:28.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:08:28.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-04-01T05:08:28.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-04-01T05:08:28.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-04-01T05:08:28.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:28.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:08:28.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-04-01T05:08:28.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-04-01T05:08:28.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-04-01T05:08:28.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:28.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:08:28.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-04-01T05:08:28.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-04-01T05:08:28.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509453' 2024-04-01T05:08:28.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:28.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-04-01T05:08:28.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:28.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:08:28.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-04-01T05:08:28.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:28.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-04-01T05:08:28.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-04-01T05:08:28.958 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182405 2024-04-01T05:08:28.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:29.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-04-01T05:08:29.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:08:30.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:08:30.240 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:30.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-04-01T05:08:30.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:30.536 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-04-01T05:08:30.536 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:30.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:08:30.537 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-04-01T05:08:30.537 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-04-01T05:08:30.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-04-01T05:08:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 128849018882 2024-04-01T05:08:30.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:08:30.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-04-01T05:08:30.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:30.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-04-01T05:08:30.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:30.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:08:30.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-04-01T05:08:30.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:30.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-04-01T05:08:30.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-01T05:08:30.824 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509453 2024-04-01T05:08:30.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:08:31.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-04-01T05:08:31.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:08:31.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:31.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:31.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:08:31.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:08:31.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:08:31.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:08:31.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:08:31.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:08:31.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:08:31.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:08:31.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:08:31.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:08:31.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:31.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:08:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:08:32.093 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:08:32.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:08:32.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:08:32.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:08:32.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:08:32.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:08:32.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:08:32.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:08:32.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:08:32.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:08:32.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:08:32.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:08:32.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-04-01T05:08:32.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:08:32.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:08:32.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:08:32.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:08:32.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:08:32.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-04-01T05:08:34.180 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:104778fc:::obj2:head# 2024-04-01T05:08:34.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:08:34.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:08:34.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:34.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:08:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:08:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:08:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:08:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:08:34.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:34.728+0000 7efe7138a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:34.731 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:34.731+0000 7efe7138a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:34.732 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:34.732+0000 7efe7138a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:35.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:08:35.012 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:08:35.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:35.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:35.254 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:35.254+0000 7efe7138a640 -1 Falling back to public interface 2024-04-01T05:08:35.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:08:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:36.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:37.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:37.536+0000 7efe7138a640 -1 osd.0 31 log_to_monitors true 2024-04-01T05:08:37.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:37.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:37.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:08:37.585 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:08:37.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:37.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/1391700008,v1:127.0.0.1:6803/1391700008] [v2:127.0.0.1:6804/1391700008,v1:127.0.0.1:6805/1391700008] exists,up 1f106366-a6fa-4b81-8beb-e7d06872eec5 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:08:37.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:08:37.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:08:37.922 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:08:37.922 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:08:37.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:08:37.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:08:37.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:08:38.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:08:38.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:08:38.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:08:38.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:08:38.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:08:38.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:08:38.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:08:38.296 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:38.296 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:08:38.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:08:38.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:38.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:08:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-04-01T05:08:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-04-01T05:08:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-04-01T05:08:38.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:38.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:08:38.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-04-01T05:08:38.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-04-01T05:08:38.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-04-01T05:08:38.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:38.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:08:38.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-04-01T05:08:38.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-04-01T05:08:38.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509456' 2024-04-01T05:08:38.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:38.612 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-04-01T05:08:38.612 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:38.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:08:38.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-04-01T05:08:38.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-04-01T05:08:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-04-01T05:08:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 150323855362 2024-04-01T05:08:38.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:38.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-04-01T05:08:38.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:08:39.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:08:39.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:40.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-04-01T05:08:40.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:40.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-04-01T05:08:40.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:40.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:08:40.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-04-01T05:08:40.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-04-01T05:08:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-04-01T05:08:40.222 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 128849018885 2024-04-01T05:08:40.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:08:40.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-04-01T05:08:40.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:40.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-04-01T05:08:40.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:40.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:08:40.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-04-01T05:08:40.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:40.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-04-01T05:08:40.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-01T05:08:40.510 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509456 2024-04-01T05:08:40.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:08:40.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-04-01T05:08:40.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:08:40.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:40.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:41.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:08:41.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:08:41.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:08:41.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:08:41.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:08:41.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:08:41.190 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:08:41.190 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:08:41.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:08:41.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:08:41.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:41.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:08:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-01T05:08:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:08:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-01T05:08:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:08:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:08:41.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:08:41.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:08:42.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:08:42.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:08:42.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:42.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:42.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:08:42.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:08:42.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:08:42.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-04-01T05:08:42.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:08:42.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:08:42.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:08:42.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:08:42.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:08:42.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:08:42.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:08:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:08:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:08:42.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:08:43.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-04-01T05:08:44.357 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:08:44.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:08:44.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:08:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:08:44.878 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:08:44.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:08:44.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:08:44.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:08:44.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:08:44.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:44.905+0000 7f87f2024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:44.932 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:44.931+0000 7f87f2024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:44.933 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:44.933+0000 7f87f2024640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:08:45.208 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:08:45.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:45.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:45.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:45.448+0000 7f87f2024640 -1 Falling back to public interface 2024-04-01T05:08:45.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:46.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:46.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:46.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:08:46.493 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:46.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:46.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:46.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:08:47.729 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:08:47.729+0000 7f87f2024640 -1 osd.1 36 log_to_monitors true 2024-04-01T05:08:47.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:08:47.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:08:47.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:08:47.788 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:08:47.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:08:47.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:08:48.146 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 40 up_thru 35 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/269078904,v1:127.0.0.1:6811/269078904] [v2:127.0.0.1:6812/269078904,v1:127.0.0.1:6813/269078904] exists,up 9313908f-bae5-412a-abdf-941d76ee6b73 2024-04-01T05:08:48.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:08:48.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:08:48.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:08:48.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:08:48.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:08:48.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:08:48.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:08:48.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:08:48.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:08:48.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:08:48.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:08:48.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:08:48.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:08:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:08:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:08:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:08:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:48.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:08:48.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-04-01T05:08:48.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-04-01T05:08:48.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-04-01T05:08:48.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:48.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:08:48.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-04-01T05:08:48.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-04-01T05:08:48.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-04-01T05:08:48.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:48.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:08:48.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-04-01T05:08:48.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-04-01T05:08:48.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509459' 2024-04-01T05:08:48.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:48.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-04-01T05:08:48.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:48.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:08:48.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-04-01T05:08:48.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:48.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-04-01T05:08:48.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-04-01T05:08:48.921 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 150323855365 2024-04-01T05:08:48.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:49.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855365 2024-04-01T05:08:49.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:08:50.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:08:50.204 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:50.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-04-01T05:08:50.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:50.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-04-01T05:08:50.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:50.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:08:50.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-04-01T05:08:50.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:50.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-04-01T05:08:50.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-04-01T05:08:50.500 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 171798691842 2024-04-01T05:08:50.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:08:50.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-04-01T05:08:50.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:50.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-04-01T05:08:50.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:50.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:08:50.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-04-01T05:08:50.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:50.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-04-01T05:08:50.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-04-01T05:08:50.785 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509459 2024-04-01T05:08:50.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:08:51.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-04-01T05:08:51.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:08:51.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:51.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:08:51.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:08:51.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:08:51.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:08:51.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:08:51.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:08:52.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:08:52.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:08:52.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:08:52.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-04-01T05:08:52.176 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0: { 2024-04-01T05:08:52.176 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-04-01T05:08:52.177 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:08:52.191 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1: { 2024-04-01T05:08:52.191 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-04-01T05:08:52.191 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:08:52.205 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2: { 2024-04-01T05:08:52.205 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-04-01T05:08:52.205 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:08:52.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-04-01T05:08:52.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T05:08:52.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-01T05:08:52.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-04-01T05:08:52.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:08:52.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-04-01T05:08:52.523 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-04-01T05:08:52.523 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:08:52.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T05:08:52.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T05:08:52.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:08:52.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-04-01T05:08:52.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-04-01T05:08:52.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:08:52.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:08:52.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:08:52.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:08:53.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:53.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-04-01T05:08:53.186 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T05:08:53.186 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": true, 2024-04-01T05:08:53.186 INFO:tasks.workunit.client.0.smithi067.stdout: "must": false, 2024-04-01T05:08:53.186 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "2024-03-18T05:07:13.186529+0000" 2024-04-01T05:08:53.186 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:08:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:08:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:08:53.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:08:02.016046+0000 '>' 2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:53.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:08:54.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:08:54.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:08:54.475 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:08:54.475 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:08:54.475 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:08:54.475 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:08:54.475 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:08:54.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:08:02.016046+0000 '>' 2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:54.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:08:55.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:08:55.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:08:55.769 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:08:55.769 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:08:55.769 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:08:55.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:08:55.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:08:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:08:02.016046+0000 '>' 2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:08:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:08:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:08:57.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:08:57.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:08:57.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:08:57.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:08:57.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:08:57.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:08:02.016046+0000 '>' 2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:57.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:08:58.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:08:58.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:08:58.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:08:58.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:08:58.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:08:58.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:08:58.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:08:54.730793+0000 '>' 2024-04-01T05:08:02.016046+0000 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:08:58.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:08:58.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:08:58.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:08:58.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:08:58.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:08:58.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:08:58.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:08:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:08:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:08:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:08:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:08:59.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:59.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:08:59.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855368 2024-04-01T05:08:59.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855368 2024-04-01T05:08:59.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368' 2024-04-01T05:08:59.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:59.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:08:59.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-04-01T05:08:59.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-04-01T05:08:59.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-04-01T05:08:59.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:08:59.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:08:59.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-04-01T05:08:59.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-04-01T05:08:59.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509462' 2024-04-01T05:08:59.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:08:59.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855368 2024-04-01T05:08:59.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:08:59.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:08:59.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855368 2024-04-01T05:08:59.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:08:59.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855368 2024-04-01T05:08:59.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-04-01T05:08:59.400 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 150323855368 2024-04-01T05:08:59.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:08:59.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855368 2024-04-01T05:08:59.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:09:00.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:09:00.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:09:01.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855368 2024-04-01T05:09:01.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:09:01.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-04-01T05:09:01.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:09:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:09:01.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-04-01T05:09:01.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:09:01.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-04-01T05:09:01.010 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 171798691845 2024-04-01T05:09:01.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-04-01T05:09:01.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:09:01.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-04-01T05:09:01.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:09:01.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-04-01T05:09:01.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:09:01.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:09:01.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-04-01T05:09:01.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:09:01.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-04-01T05:09:01.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-04-01T05:09:01.299 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509462 2024-04-01T05:09:01.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:09:01.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-04-01T05:09:01.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:09:01.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:09:01.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:09:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:09:01.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:09:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:09:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:09:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:09:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:09:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:09:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:09:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:09:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-04-01T05:09:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:09:02.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:09:02.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:09:02.849 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:09:02.849 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:09:02.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:09:02.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:09:02.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:09:02.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-04-01T05:09:02.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-04-01T05:09:02.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370' 2024-04-01T05:09:02.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:09:02.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:09:03.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691847 2024-04-01T05:09:03.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691847 2024-04-01T05:09:03.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-04-01T05:09:03.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:09:03.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:09:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-04-01T05:09:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-04-01T05:09:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509464' 2024-04-01T05:09:03.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:09:03.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855370 2024-04-01T05:09:03.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:09:03.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:09:03.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855370 2024-04-01T05:09:03.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:09:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-04-01T05:09:03.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-04-01T05:09:03.167 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 150323855370 2024-04-01T05:09:03.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:09:03.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855370 2024-04-01T05:09:03.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:09:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:09:04.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:09:04.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-04-01T05:09:04.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:09:04.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691847 2024-04-01T05:09:04.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:09:04.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:09:04.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691847 2024-04-01T05:09:04.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:09:04.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691847 2024-04-01T05:09:04.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-04-01T05:09:04.747 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 171798691847 2024-04-01T05:09:04.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:09:05.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691847 2024-04-01T05:09:05.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:09:05.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-04-01T05:09:05.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:09:05.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:09:05.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-04-01T05:09:05.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:09:05.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-04-01T05:09:05.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-04-01T05:09:05.035 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509464 2024-04-01T05:09:05.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:09:05.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-04-01T05:09:05.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-04-01T05:09:05.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-04-01T05:09:05.582 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:09:05.582 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-04-01T05:08:54.730903+0000 21'10 41:92 [1,0] 1 [1,0] 1 21'10 2024-04-01T05:08:54.730793+0000 21'10 2024-04-01T05:08:54.730793+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:08:54.730793+0000 10 0 2024-04-01T05:09:05.582 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:09:05.582 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:09:05.583 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:09:05.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-04-01T05:09:05.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-04-01T05:09:05.858 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:09:05.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:09:05.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:09:05.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:09:05.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:09:05.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:09:05.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:09:05.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:09:05.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:09:05.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:09:05.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:09:05.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:09:05.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:09:05.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:09:05.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:09:05.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:09:05.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:09:05.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:09:05.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:09:05.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:09:05.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:09:05.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:09:05.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:09:05.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:09:05.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:09:05.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:05.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:05.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:09:06.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:09:06.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:09:06.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:09:06.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:09:06.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:09:06.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:09:06.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:09:06.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:09:06.003 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:09:06.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:09:06.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:09:06.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:09:06.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:09:06.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:09:06.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:09:06.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:09:06.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:09:06.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:09:06.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:09:06.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:09:06.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:06.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:06.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:09:06.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:09:06.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:09:06.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:09:06.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:09:06.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:06.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:06.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:09:06.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:09:06.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:09:06.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:09:06.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:09:06.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:09:06.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:09:06.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:09:06.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:09:06.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:09:06.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:09:06.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:09:06.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:09:06.571 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:09:06.571 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:09:06.571 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:09:06.571 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:09:06.572 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:06.572 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:06.572 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:09:06.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:09:06.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:09:06.657 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:09:06.658 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:09:06.658 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:06.658 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:06.658 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:09:06.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:09:06.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:09:06.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-04-01T05:09:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:09:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:09:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:09:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:09:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:09:06.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:09:07.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:09:07.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:09:07.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-04-01T05:09:07.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-04-01T05:09:07.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-04-01T05:09:07.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:09:07.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:09:07.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:09:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a350249-afd0-4450-ae2c-721418e6ba7b 2024-04-01T05:09:07.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2a350249-afd0-4450-ae2c-721418e6ba7b' 2024-04-01T05:09:07.593 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 2a350249-afd0-4450-ae2c-721418e6ba7b 2024-04-01T05:09:07.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:09:07.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzQQpmPcM6JBAAY1BKTsqGqWfM1FdkznUGwA== 2024-04-01T05:09:07.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzQQpmPcM6JBAAY1BKTsqGqWfM1FdkznUGwA=="}' 2024-04-01T05:09:07.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a350249-afd0-4450-ae2c-721418e6ba7b -i td/osd-scrub-repair/0/new.json 2024-04-01T05:09:07.819 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:09:07.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:09:07.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBzQQpmPcM6JBAAY1BKTsqGqWfM1FdkznUGwA== --osd-uuid 2a350249-afd0-4450-ae2c-721418e6ba7b 2024-04-01T05:09:07.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.864+0000 7f101e4d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:07.870 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.870+0000 7f101e4d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:07.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.871+0000 7f101e4d1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:07.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.871+0000 7f101e4d1640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:09:07.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.871+0000 7f101e4d1640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:09:07.872 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.871+0000 7f101e4d1640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:09:07.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:07.962+0000 7f101e4d1640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:09:13.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:09:13.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:09:13.478 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:09:13.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:09:13.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:09:13.849 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:09:13.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:09:13.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:09:13.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:09:13.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:09:13.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:09:13.878 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:13.878+0000 7f5171c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:13.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:13.879+0000 7f5171c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:13.881 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:13.881+0000 7f5171c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:09:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:09:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:09:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:09:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:09:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:14.150 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:09:14.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:09:14.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:14.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:14.408 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:14.408+0000 7f5171c13640 -1 Falling back to public interface 2024-04-01T05:09:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:15.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:15.429 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:09:15.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:15.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:09:15.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:15.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:15.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:16.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:16.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:16.722 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:09:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:09:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:16.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:16.727 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:16.727+0000 7f5171c13640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:09:17.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:18.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:18.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:18.024 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:09:18.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:09:18.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:18.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:18.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:19.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:19.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:19.319 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:09:19.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:09:19.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:19.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:19.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:20.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:20.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:20.691 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:09:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:09:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:20.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:20.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:21.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:21.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:21.986 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:09:21.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:09:21.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:21.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:23.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:23.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:23.324 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:09:23.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:09:23.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:23.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:23.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:24.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:24.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:24.677 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:09:24.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:09:24.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:24.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:24.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:25.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:25.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:25.969 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:09:25.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:09:25.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:25.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:26.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:27.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:27.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:27.295 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:09:27.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:09:27.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:27.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:27.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:28.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:28.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:28.591 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:09:28.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:09:28.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:28.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:29.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:30.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:30.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:30.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:09:30.179 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:09:30.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:30.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:30.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:31.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:31.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:31.921 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:09:31.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:09:31.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:31.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:32.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:33.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:33.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:33.828 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:09:33.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:09:33.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:33.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:34.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:35.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:35.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:35.286 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:09:35.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:09:35.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:35.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:35.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:36.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:36.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:09:36.978 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:09:36.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:36.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:37.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:38.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:38.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:38.695 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:09:38.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:09:38.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:38.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:09:39.002 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3540224373,v1:127.0.0.1:6803/3540224373] [v2:127.0.0.1:6804/3540224373,v1:127.0.0.1:6805/3540224373] exists,up 2a350249-afd0-4450-ae2c-721418e6ba7b 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:09:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:09:39.007 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-04-01T05:09:39.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:09:39.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:09:39.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=73df97e7-4475-4f69-a121-b2aecadb9310 2024-04-01T05:09:39.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 73df97e7-4475-4f69-a121-b2aecadb9310' 2024-04-01T05:09:39.009 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 73df97e7-4475-4f69-a121-b2aecadb9310 2024-04-01T05:09:39.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:09:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTQQpmTKliARAAhdmE8EoxM9mqREQclJt2yA== 2024-04-01T05:09:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTQQpmTKliARAAhdmE8EoxM9mqREQclJt2yA=="}' 2024-04-01T05:09:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 73df97e7-4475-4f69-a121-b2aecadb9310 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:09:39.364 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:09:39.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:09:39.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCTQQpmTKliARAAhdmE8EoxM9mqREQclJt2yA== --osd-uuid 73df97e7-4475-4f69-a121-b2aecadb9310 2024-04-01T05:09:39.407 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.406+0000 7f0faf2c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:39.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.448+0000 7f0faf2c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.449+0000 7f0faf2c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.449+0000 7f0faf2c4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:09:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.449+0000 7f0faf2c4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:09:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.449+0000 7f0faf2c4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:09:39.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:39.523+0000 7f0faf2c4640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:09:45.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:09:45.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:09:45.265 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:09:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:09:45.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:09:45.631 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:09:45.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:09:45.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:09:45.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:09:45.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:09:45.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:09:45.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:45.660+0000 7f4eb180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:45.662 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:45.662+0000 7f4eb180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:45.663 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:45.663+0000 7f4eb180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:09:45.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:09:45.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:09:45.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:09:45.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:09:45.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:09:45.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:45.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:09:45.915 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:09:45.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:45.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:46.182 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:46.182+0000 7f4eb180b640 -1 Falling back to public interface 2024-04-01T05:09:46.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:47.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:47.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:47.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:09:47.213 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:09:47.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:47.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:47.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:48.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:48.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:48.507 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:09:48.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:09:48.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:48.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:48.508 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:09:48.508+0000 7f4eb180b640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:09:48.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:49.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:49.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:49.826 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:09:49.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:09:49.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:49.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:50.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:51.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:51.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:51.208 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:09:51.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:09:51.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:51.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:51.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:52.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:52.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:52.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:09:52.506 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:09:52.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:52.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:53.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:53.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:53.919 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:09:53.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:09:53.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:53.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:54.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:55.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:55.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:55.259 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:09:55.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:09:55.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:55.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:55.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:56.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:56.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:56.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:09:56.551 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:09:56.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:56.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:56.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:57.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:57.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:57.888 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:09:57.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:09:57.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:57.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:58.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:09:59.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:09:59.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:09:59.180 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:09:59.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:09:59.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:09:59.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:09:59.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:00.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:00.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:00.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:10:00.476 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:10:00.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:00.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:00.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:01.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:01.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:01.976 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:10:01.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:10:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:10:03.669 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:10:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:03.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:04.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:04.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:04.996 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:10:04.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:10:04.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:04.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:05.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:06.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:06.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:06.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:10:06.667 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:10:06.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:06.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:07.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:08.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:08.310 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:10:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:10:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:08.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:09.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:10.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:10.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:10.080 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:10:10.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:10:10.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:10.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:10.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:11.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:11.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:11.499 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:10:11.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:10:11.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:11.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:10:11.802 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3761711346,v1:127.0.0.1:6811/3761711346] [v2:127.0.0.1:6812/3761711346,v1:127.0.0.1:6813/3761711346] exists,up 73df97e7-4475-4f69-a121-b2aecadb9310 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:10:11.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:10:11.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:10:11.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=48242c7d-c73d-4a5a-a14e-a3746d0558d8 2024-04-01T05:10:11.809 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 48242c7d-c73d-4a5a-a14e-a3746d0558d8 2024-04-01T05:10:11.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 48242c7d-c73d-4a5a-a14e-a3746d0558d8' 2024-04-01T05:10:11.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:10:11.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCzQQpmkfMbMRAASN21SVu1CC1Ljx4vFFM8WA== 2024-04-01T05:10:11.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCzQQpmkfMbMRAASN21SVu1CC1Ljx4vFFM8WA=="}' 2024-04-01T05:10:11.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 48242c7d-c73d-4a5a-a14e-a3746d0558d8 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:10:12.168 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:10:12.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:10:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCzQQpmkfMbMRAASN21SVu1CC1Ljx4vFFM8WA== --osd-uuid 48242c7d-c73d-4a5a-a14e-a3746d0558d8 2024-04-01T05:10:12.211 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.210+0000 7fe5ce7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:12.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.261+0000 7fe5ce7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:12.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.262+0000 7fe5ce7bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:12.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.262+0000 7fe5ce7bb640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:10:12.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.262+0000 7fe5ce7bb640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:10:12.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.262+0000 7fe5ce7bb640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:10:12.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:12.336+0000 7fe5ce7bb640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:10:18.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:10:18.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:10:18.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:10:18.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:10:18.631 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:10:18.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:10:18.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:10:18.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:10:18.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:10:18.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:10:18.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:18.658+0000 7fce128ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:18.676 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:18.675+0000 7fce128ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:18.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:18.676+0000 7fce128ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:18.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:10:18.952 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:10:18.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:18.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:19.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:19.204+0000 7fce128ae640 -1 Falling back to public interface 2024-04-01T05:10:19.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:20.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:20.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:20.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:10:20.234 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:10:20.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:20.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:20.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:21.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:21.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:21.525 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:10:21.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:10:21.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:21.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:21.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:21.527+0000 7fce128ae640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:10:21.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:22.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:22.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:22.837 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:10:22.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:10:22.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:22.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:23.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:24.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:24.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:24.169 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:10:24.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:10:24.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:24.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:24.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:25.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:25.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:25.503 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:10:25.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:10:25.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:25.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:25.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:26.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:26.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:26.855 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:10:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:10:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:26.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:27.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:28.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:28.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:28.181 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:10:28.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:10:28.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:28.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:28.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:29.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:29.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:29.519 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:10:29.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:10:29.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:29.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:29.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:30.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:30.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:30.809 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:10:30.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:10:30.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:30.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:31.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:32.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:32.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:32.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:10:32.158 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:10:32.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:32.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:32.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:33.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:33.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:33.524 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:10:33.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:10:33.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:33.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:33.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:34.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:34.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:34.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:10:34.931 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:10:34.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:34.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:35.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:36.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:36.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:36.536 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:10:36.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:10:36.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:36.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:36.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:37.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:37.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:37.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:10:37.836 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:10:37.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:37.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:38.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:39.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:39.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:39.155 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:10:39.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:10:39.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:39.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:39.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:40.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:40.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:40.746 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:10:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:10:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:41.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:42.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:42.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:42.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:10:42.513 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:10:42.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:42.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:43.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:43.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:43.846 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:10:43.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:10:43.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:43.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:44.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:45.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:45.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:45.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:10:45.192 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:10:45.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:45.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:10:45.509 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2060902380,v1:127.0.0.1:6819/2060902380] [v2:127.0.0.1:6820/2060902380,v1:127.0.0.1:6821/2060902380] exists,up 48242c7d-c73d-4a5a-a14e-a3746d0558d8 2024-04-01T05:10:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:10:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:10:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:10:45.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-04-01T05:10:45.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-04-01T05:10:45.968 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T05:10:45.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:10:46.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-04-01T05:10:47.418 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 2 2024-04-01T05:10:47.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-04-01T05:10:47.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:10:47.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:10:47.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:10:47.430 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:10:47.431 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:10:47.431 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:10:47.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:10:47.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:10:47.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:10:47.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:10:47.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:10:47.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:10:47.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:10:47.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:10:47.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:10:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:10:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:10:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:10:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:10:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:10:48.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:10:48.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-04-01T05:10:48.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-04-01T05:10:48.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-04-01T05:10:48.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:10:48.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:10:48.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-04-01T05:10:48.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-04-01T05:10:48.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2024-04-01T05:10:48.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:10:48.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:10:48.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-04-01T05:10:48.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-04-01T05:10:48.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509451' 2024-04-01T05:10:48.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:10:48.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-04-01T05:10:48.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:10:48.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:10:48.422 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-04-01T05:10:48.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:10:48.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-04-01T05:10:48.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-04-01T05:10:48.424 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836495 2024-04-01T05:10:48.424 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:10:48.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-04-01T05:10:48.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:10:49.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:10:49.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:10:50.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836495 2024-04-01T05:10:50.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:10:50.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-04-01T05:10:50.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:10:50.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:10:50.003 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-04-01T05:10:50.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:10:50.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-04-01T05:10:50.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-04-01T05:10:50.008 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672974 2024-04-01T05:10:50.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:10:50.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-04-01T05:10:50.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:10:50.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-04-01T05:10:50.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:10:50.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:10:50.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-04-01T05:10:50.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:10:50.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-04-01T05:10:50.293 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509451 2024-04-01T05:10:50.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-01T05:10:50.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:10:50.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-01T05:10:50.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:10:50.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:10:50.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:10:50.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:10:50.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:10:50.970 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:10:50.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:10:50.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:10:50.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:10:50.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:10:50.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:10:51.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-01T05:10:51.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:10:51.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:10:51.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:10:51.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-01T05:10:51.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-01T05:10:51.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:10:51.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-01T05:10:51.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:10:51.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:10:51.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:10:51.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:10:51.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:10:51.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:10:51.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:10:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:10:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:10:52.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:10:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-04-01T05:10:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-04-01T05:10:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:10:52.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:10:52.551 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:10:52.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:10:52.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:10:52.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:10:53.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:10:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:10:53.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:10:53.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:10:53.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-04-01T05:10:54.775 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-04-01T05:10:55.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:10:55.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:10:55.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:10:55.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:10:55.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-04-01T05:10:55.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:10:55.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:10:55.314 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:10:55.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:10:55.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:10:55.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:10:55.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:10:55.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:10:55.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:10:55.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:55.325+0000 7f5a4e224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:55.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:55.328+0000 7f5a4e224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:55.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:55.329+0000 7f5a4e224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:10:55.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:55.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:10:55.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:55.847+0000 7f5a4e224640 -1 Falling back to public interface 2024-04-01T05:10:55.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:56.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:56.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:56.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:10:56.882 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:10:56.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:56.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:10:57.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:58.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:58.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:58.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:10:58.175 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:10:58.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:58.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:10:58.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:10:58.340+0000 7f5a4e224640 -1 osd.0 21 log_to_monitors true 2024-04-01T05:10:58.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:10:59.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:10:59.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:10:59.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:10:59.554 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:10:59.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:10:59.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:10:59.849 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4124285217,v1:127.0.0.1:6803/4124285217] [v2:127.0.0.1:6804/4124285217,v1:127.0.0.1:6805/4124285217] exists,up 2a350249-afd0-4450-ae2c-721418e6ba7b 2024-04-01T05:10:59.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:10:59.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:10:59.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:10:59.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:10:59.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:10:59.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:10:59.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:10:59.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:11:00.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:11:00.212 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:11:00.212 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:11:00.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:11:00.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:00.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:11:00.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-04-01T05:11:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-04-01T05:11:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-04-01T05:11:00.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:00.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:11:00.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-04-01T05:11:00.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-04-01T05:11:00.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-04-01T05:11:00.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:00.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:11:00.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-04-01T05:11:00.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-04-01T05:11:00.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509455' 2024-04-01T05:11:00.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:00.528 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-04-01T05:11:00.528 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:00.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:11:00.530 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-04-01T05:11:00.530 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:00.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-04-01T05:11:00.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-04-01T05:11:00.531 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182402 2024-04-01T05:11:00.531 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:11:00.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-04-01T05:11:00.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:11:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:11:01.817 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:11:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-04-01T05:11:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:02.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-04-01T05:11:02.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:11:02.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-04-01T05:11:02.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-04-01T05:11:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-01T05:11:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672977 2024-04-01T05:11:02.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:11:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-04-01T05:11:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:02.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-04-01T05:11:02.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:11:02.411 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-04-01T05:11:02.411 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-04-01T05:11:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-01T05:11:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509455 2024-04-01T05:11:02.412 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:11:02.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-01T05:11:02.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:11:02.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:11:02.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:11:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:11:03.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:11:03.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:11:03.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:11:03.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:11:03.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:11:03.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:11:03.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:11:03.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:11:03.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:11:03.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:11:03.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-04-01T05:11:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:11:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-04-01T05:11:03.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-04-01T05:11:03.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:11:03.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:11:03.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:11:03.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:11:04.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-04-01T05:11:04.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-04-01T05:11:04.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:11:04.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:11:04.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:11:04.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:11:04.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:04.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-04-01T05:11:04.702 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T05:11:04.702 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": false, 2024-04-01T05:11:04.702 INFO:tasks.workunit.client.0.smithi067.stdout: "must": false, 2024-04-01T05:11:04.702 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "2024-03-25T05:09:24.702884+0000" 2024-04-01T05:11:04.702 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:11:04.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:04.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:11:04.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:11:04.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:10:45.909368+0000 '>' 2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:04.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:11:05.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:11:05.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:11:05.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:11:05.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:11:05.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:11:05.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:11:05.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:11:06.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:10:45.909368+0000 '>' 2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:06.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:11:07.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:11:07.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:11:07.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:11:07.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:11:07.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:11:07.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:11:07.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:11:07.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:10:45.909368+0000 '>' 2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:07.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:11:08.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:11:08.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:11:08.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T05:11:08.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:11:08.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:11:08.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:11:08.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:11:08.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:11:05.702447+0000 '>' 2024-04-01T05:10:45.909368+0000 2024-04-01T05:11:08.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:11:08.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-04-01T05:11:09.156 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:11:09.156 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-04-01T05:11:05.893420+0000 21'1 26:59 [1,0] 1 [1,0] 1 21'1 2024-04-01T05:11:05.702447+0000 21'1 2024-04-01T05:11:05.702447+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:11:05.702447+0000 1 0 2024-04-01T05:11:09.156 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:11:09.157 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:11:09.157 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:11:09.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-04-01T05:11:11.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-04-01T05:11:11.445 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:11:11.445 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-04-01T05:11:05.893420+0000 21'1 26:59 [1,0] 1 [1,0] 1 21'1 2024-04-01T05:11:05.702447+0000 21'1 2024-04-01T05:11:05.702447+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:11:05.702447+0000 1 0 2024-04-01T05:11:11.445 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:11:11.445 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:11:11.446 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:11:11.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-04-01T05:11:13.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-04-01T05:11:13.736 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:11:13.736 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-04-01T05:11:05.893420+0000 21'1 26:59 [1,0] 1 [1,0] 1 21'1 2024-04-01T05:11:05.702447+0000 21'1 2024-04-01T05:11:05.702447+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:11:05.702447+0000 1 0 2024-04-01T05:11:13.736 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:11:13.736 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:11:13.736 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:11:13.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-04-01T05:11:18.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-04-01T05:11:18.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:11:18.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:11:18.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:11:18.751 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:11:18.751 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:11:18.751 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:11:18.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:11:18.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:11:18.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:11:18.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:11:18.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:11:18.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:11:18.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:11:18.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:11:18.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:11:19.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:11:19.128 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:11:19.128 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:11:19.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:11:19.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:19.129 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:11:19.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-04-01T05:11:19.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-04-01T05:11:19.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-04-01T05:11:19.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:19.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:11:19.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-04-01T05:11:19.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-04-01T05:11:19.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672982' 2024-04-01T05:11:19.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:19.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:11:19.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-04-01T05:11:19.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-04-01T05:11:19.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672982 2-64424509459' 2024-04-01T05:11:19.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:19.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-04-01T05:11:19.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:19.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:11:19.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-04-01T05:11:19.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:19.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-04-01T05:11:19.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-04-01T05:11:19.447 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 107374182407 2024-04-01T05:11:19.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:11:19.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182407 2024-04-01T05:11:19.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:11:20.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:11:20.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:11:21.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-04-01T05:11:21.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:21.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-04-01T05:11:21.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:21.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:11:21.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-04-01T05:11:21.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:21.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-04-01T05:11:21.028 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672982 2024-04-01T05:11:21.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-01T05:11:21.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:11:21.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-01T05:11:21.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:21.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-04-01T05:11:21.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:21.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:11:21.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-04-01T05:11:21.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:21.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-04-01T05:11:21.317 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509459 2024-04-01T05:11:21.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-04-01T05:11:21.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:11:21.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-04-01T05:11:21.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:11:21.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:11:21.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:11:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:11:21.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:11:22.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:11:22.240 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:11:22.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:11:22.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:11:22.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:11:22.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:11:22.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:11:22.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-04-01T05:11:22.864 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:11:22.864 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-04-01T05:11:05.893420+0000 21'1 26:59 [1,0] 1 [1,0] 1 21'1 2024-04-01T05:11:05.702447+0000 21'1 2024-04-01T05:11:05.702447+0000 0 1 periodic scrub scheduled @ 2024-04-02T05:11:05.702447+0000 1 0 2024-04-01T05:11:22.864 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:11:22.864 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:11:22.864 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:11:22.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-04-01T05:11:22.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:11:22.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:11:22.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:11:22.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:11:22.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:11:22.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:11:22.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:11:23.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:11:23.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:11:23.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:11:23.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:11:23.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:11:23.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-04-01T05:11:24.966 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:11:24.966 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:11:25.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:11:25.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:11:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-04-01T05:11:25.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:11:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:11:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:11:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:11:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:11:25.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:25.270+0000 7f2557a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:25.272 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:25.272+0000 7f2557a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:25.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:25.274+0000 7f2557a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:25.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:11:25.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:11:25.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:25.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:25.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:25.786+0000 7f2557a0d640 -1 Falling back to public interface 2024-04-01T05:11:25.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:11:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:11:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:27.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:28.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:28.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:28.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:11:28.112 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:11:28.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:28.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:28.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:28.174+0000 7f2557a0d640 -1 osd.0 26 log_to_monitors true 2024-04-01T05:11:28.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:29.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:29.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:29.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:11:29.408 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:11:29.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:29.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/356268614,v1:127.0.0.1:6803/356268614] [v2:127.0.0.1:6804/356268614,v1:127.0.0.1:6805/356268614] exists,up 2a350249-afd0-4450-ae2c-721418e6ba7b 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:11:29.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:11:29.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:11:29.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:11:29.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:11:29.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:11:29.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:11:29.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:11:29.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:11:30.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:11:30.065 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:11:30.065 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:11:30.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:11:30.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:30.065 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:11:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-04-01T05:11:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-04-01T05:11:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-04-01T05:11:30.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:30.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:11:30.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-04-01T05:11:30.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-04-01T05:11:30.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-04-01T05:11:30.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:11:30.275 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:11:30.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-04-01T05:11:30.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-04-01T05:11:30.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509463' 2024-04-01T05:11:30.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:30.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-04-01T05:11:30.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:30.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:11:30.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-04-01T05:11:30.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:30.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-04-01T05:11:30.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-04-01T05:11:30.382 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 128849018882 2024-04-01T05:11:30.382 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:11:30.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-04-01T05:11:30.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:11:31.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:11:31.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:11:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-04-01T05:11:31.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:31.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-04-01T05:11:31.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:31.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:11:31.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-04-01T05:11:31.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-04-01T05:11:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-01T05:11:31.960 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672985 2024-04-01T05:11:31.960 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:11:32.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672985 2024-04-01T05:11:32.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:11:32.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-04-01T05:11:32.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:11:32.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:11:32.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-04-01T05:11:32.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:11:32.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-04-01T05:11:32.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-04-01T05:11:32.256 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509463 2024-04-01T05:11:32.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:11:32.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-04-01T05:11:32.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:11:32.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:11:32.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:11:32.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:11:33.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:11:33.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:11:33.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:11:33.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:11:33.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:11:33.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:11:33.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:11:33.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:11:33.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:11:33.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:04.736+0000 7f4e95784640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:04.736+0000 7f4e95784640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:04.736+0000 7f4e95784640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:05.700+0000 7f4e95784640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:05.700+0000 7f4e95784640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:05.700+0000 7f4e95784640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:05.700+0000 7f4e95784640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-04-01T05:11:33.580 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:11:05.700+0000 7f4e95784640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:11:33.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-04-01T05:11:33.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:11:33.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:11:33.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:11:33.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:11:33.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:11:33.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:11:33.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:11:33.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:11:33.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:11:33.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:11:33.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:11:33.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:11:33.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:11:33.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:11:33.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:11:33.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:11:33.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:11:33.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:11:33.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:33.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:33.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:11:33.841 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:11:33.842 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:11:33.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:11:33.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:11:33.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:11:33.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:11:33.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:11:33.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:11:33.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:11:33.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:11:33.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:11:33.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:11:33.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:11:33.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:11:33.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:11:33.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:11:33.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:11:33.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:11:33.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:11:33.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:11:33.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:33.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:33.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:11:33.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:11:33.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:11:33.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:11:33.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:11:33.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:33.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:33.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:11:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:11:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:11:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:11:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:34.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:11:34.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:11:34.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:11:34.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:11:34.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:11:34.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:11:34.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:11:34.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:11:34.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:11:34.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:11:34.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-04-01T05:11:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:11:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:11:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:11:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:11:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:11:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:11:35.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:35.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:11:35.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:11:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:11:35.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-04-01T05:11:35.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:11:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:11:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:11:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:11:35.409 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:11:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 477b6684-b468-45fb-a574-a334ef8353ed' 2024-04-01T05:11:35.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:11:35.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHQgpm/GtHGRAAGx1Y8bBIrMUs5SyflIbggQ== 2024-04-01T05:11:35.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHQgpm/GtHGRAAGx1Y8bBIrMUs5SyflIbggQ=="}' 2024-04-01T05:11:35.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 477b6684-b468-45fb-a574-a334ef8353ed -i td/osd-scrub-repair/0/new.json 2024-04-01T05:11:35.629 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:11:35.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:11:35.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAHQgpm/GtHGRAAGx1Y8bBIrMUs5SyflIbggQ== --osd-uuid 477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:11:35.671 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.671+0000 7f93cfe12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:35.680 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.680+0000 7f93cfe12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:35.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.681+0000 7f93cfe12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:35.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.681+0000 7f93cfe12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:11:35.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.681+0000 7f93cfe12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:11:35.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.681+0000 7f93cfe12640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:11:35.764 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:35.763+0000 7f93cfe12640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:11:41.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:11:41.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:11:41.204 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:11:41.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:11:41.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:11:41.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:11:41.615 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:11:41.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:11:41.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:11:41.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:11:41.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:11:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:41.644+0000 7faf7e349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:41.646+0000 7faf7e349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:41.647+0000 7faf7e349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:41.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:11:41.902 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:11:41.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:41.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:42.160 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:42.160+0000 7faf7e349640 -1 Falling back to public interface 2024-04-01T05:11:42.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:43.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:43.187 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:11:43.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:43.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:11:43.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:43.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:43.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:44.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:44.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:44.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:11:44.484 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:11:44.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:44.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:44.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:44.503+0000 7faf7e349640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:11:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:45.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:45.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:45.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:11:45.805 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:11:45.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:45.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:11:46.097 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3148299083,v1:127.0.0.1:6803/3148299083] [v2:127.0.0.1:6804/3148299083,v1:127.0.0.1:6805/3148299083] exists,up 477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:11:46.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:11:46.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:11:46.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:11:46.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:11:46.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:11:46.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=825caaa9-cb5a-4b2f-b9b3-ce42f7b36b3d 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 825caaa9-cb5a-4b2f-b9b3-ce42f7b36b3d' 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 825caaa9-cb5a-4b2f-b9b3-ce42f7b36b3d 2024-04-01T05:11:46.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:11:46.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASQgpmZcgPBxAAMvQklTIH2XKONUm7ZsH9gg== 2024-04-01T05:11:46.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASQgpmZcgPBxAAMvQklTIH2XKONUm7ZsH9gg=="}' 2024-04-01T05:11:46.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 825caaa9-cb5a-4b2f-b9b3-ce42f7b36b3d -i td/osd-scrub-repair/1/new.json 2024-04-01T05:11:46.482 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:11:46.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:11:46.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQASQgpmZcgPBxAAMvQklTIH2XKONUm7ZsH9gg== --osd-uuid 825caaa9-cb5a-4b2f-b9b3-ce42f7b36b3d 2024-04-01T05:11:46.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.524+0000 7f374c0d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:46.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.566+0000 7f374c0d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:46.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.567+0000 7f374c0d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:46.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.567+0000 7f374c0d7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:11:46.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.567+0000 7f374c0d7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:11:46.568 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.567+0000 7f374c0d7640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:11:46.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:46.649+0000 7f374c0d7640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:11:52.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:11:52.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:11:52.256 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:11:52.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:11:52.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:11:52.625 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:11:52.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:11:52.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:11:52.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:11:52.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:11:52.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:11:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:52.653+0000 7f0619316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:52.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:52.655+0000 7f0619316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:52.656+0000 7f0619316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:11:52.949 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:11:52.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:52.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:11:53.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:53.173+0000 7f0619316640 -1 Falling back to public interface 2024-04-01T05:11:53.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:54.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:54.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:54.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:11:54.273 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:11:54.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:54.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:11:54.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:55.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:55.483+0000 7f0619316640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:11:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:11:55.564 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:11:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:55.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:11:55.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:11:56.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:11:56.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:11:56.885 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:11:56.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:11:56.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:11:56.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:11:57.176 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/229044267,v1:127.0.0.1:6811/229044267] [v2:127.0.0.1:6812/229044267,v1:127.0.0.1:6813/229044267] exists,up 825caaa9-cb5a-4b2f-b9b3-ce42f7b36b3d 2024-04-01T05:11:57.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:11:57.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:11:57.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:11:57.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:11:57.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37c43dbb-d3b2-49d1-82ee-e9f020e10982 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 37c43dbb-d3b2-49d1-82ee-e9f020e10982' 2024-04-01T05:11:57.183 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 37c43dbb-d3b2-49d1-82ee-e9f020e10982 2024-04-01T05:11:57.184 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:11:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdQgpmYu/ICxAAlhDlEyYCQVSzzOJg4q4+VQ== 2024-04-01T05:11:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdQgpmYu/ICxAAlhDlEyYCQVSzzOJg4q4+VQ=="}' 2024-04-01T05:11:57.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37c43dbb-d3b2-49d1-82ee-e9f020e10982 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:11:57.562 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:11:57.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:11:57.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAdQgpmYu/ICxAAlhDlEyYCQVSzzOJg4q4+VQ== --osd-uuid 37c43dbb-d3b2-49d1-82ee-e9f020e10982 2024-04-01T05:11:57.604 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.604+0000 7fc46e220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:57.650 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.650+0000 7fc46e220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:57.651 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.651+0000 7fc46e220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:11:57.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.652+0000 7fc46e220640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:11:57.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.652+0000 7fc46e220640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:11:57.652 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.652+0000 7fc46e220640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:11:57.738 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:11:57.738+0000 7fc46e220640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:12:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:12:03.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:12:03.463 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:12:03.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:12:03.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:12:03.834 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:12:03.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:12:03.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:12:03.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:12:03.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:12:03.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:12:03.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:03.863+0000 7f9b6ae12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:03.865 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:03.865+0000 7f9b6ae12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:03.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:03.866+0000 7f9b6ae12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:04.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:12:04.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:12:04.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:12:04.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:12:04.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:12:04.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:04.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:12:04.152 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:12:04.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:04.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:12:04.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:04.388+0000 7f9b6ae12640 -1 Falling back to public interface 2024-04-01T05:12:04.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:05.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:05.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:05.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:12:05.451 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:12:05.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:05.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:12:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:06.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:06.746 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:12:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:12:06.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:06.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:12:06.799 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:06.799+0000 7f9b6ae12640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:12:07.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:08.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:08.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:08.040 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:12:08.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:12:08.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:08.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:12:08.331 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/982952741,v1:127.0.0.1:6819/982952741] [v2:127.0.0.1:6820/982952741,v1:127.0.0.1:6821/982952741] exists,up 37c43dbb-d3b2-49d1-82ee-e9f020e10982 2024-04-01T05:12:08.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:12:08.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:12:08.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:12:08.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-04-01T05:12:08.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-04-01T05:12:08.710 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T05:12:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:12:09.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-04-01T05:12:10.320 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 2 2024-04-01T05:12:10.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-04-01T05:12:10.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:12:10.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:12:10.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:12:10.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:12:10.333 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:12:10.333 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:12:10.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:12:10.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:12:10.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:12:10.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:12:10.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:12:10.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:12:10.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:12:10.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:12:10.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:12:10.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:12:10.715 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:10.715 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:12:10.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:12:10.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:10.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:12:10.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-04-01T05:12:10.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-04-01T05:12:10.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-04-01T05:12:10.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:10.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:12:11.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-04-01T05:12:11.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-04-01T05:12:11.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-04-01T05:12:11.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:11.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:12:11.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-04-01T05:12:11.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-04-01T05:12:11.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-04-01T05:12:11.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:11.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-04-01T05:12:11.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:11.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:12:11.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-04-01T05:12:11.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:11.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-04-01T05:12:11.132 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836487 2024-04-01T05:12:11.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-04-01T05:12:11.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:11.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-04-01T05:12:11.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:12:12.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:12:12.438 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:12.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-04-01T05:12:12.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:12.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-04-01T05:12:12.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:12.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:12:12.761 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-04-01T05:12:12.761 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:12.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-04-01T05:12:12.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-04-01T05:12:12.763 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672964 2024-04-01T05:12:12.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:12:13.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-04-01T05:12:13.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:13.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-04-01T05:12:13.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:13.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:12:13.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-04-01T05:12:13.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:13.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-04-01T05:12:13.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-04-01T05:12:13.051 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509442 2024-04-01T05:12:13.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:12:13.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-04-01T05:12:13.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:12:13.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:12:13.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:12:13.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:12:13.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:12:14.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:12:14.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:12:14.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:12:14.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:12:14.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:12:14.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:12:14.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-04-01T05:12:14.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-04-01T05:12:14.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:12:14.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-04-01T05:12:14.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:12:14.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:12:14.492 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:12:14.493 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:12:14.493 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:12:14.493 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:14.493 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:12:14.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:12:14.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:14.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:12:15.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:12:15.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:12:15.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:12:15.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:12:15.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:12:15.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-04-01T05:12:16.817 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-04-01T05:12:17.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:12:17.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:12:17.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:12:17.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:12:17.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:12:17.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:12:17.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:12:17.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:12:17.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:12:17.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:12:17.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:12:17.347 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:12:17.348 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:12:17.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:12:17.349 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:12:17.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:12:17.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:12:17.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:12:17.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:12:17.376 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:17.375+0000 7f5398019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:17.378 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:17.378+0000 7f5398019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:17.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:17.379+0000 7f5398019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:17.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:12:17.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:12:17.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:12:17.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:12:17.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:12:17.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:17.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:12:17.641 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:12:17.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:17.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:17.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:17.899+0000 7f5398019640 -1 Falling back to public interface 2024-04-01T05:12:17.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:18.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:18.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:18.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:12:18.920 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:18.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:18.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:19.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:20.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:20.190+0000 7f5398019640 -1 osd.0 21 log_to_monitors true 2024-04-01T05:12:20.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:20.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:20.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:12:20.216 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:12:20.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:20.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:20.639 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2999687971,v1:127.0.0.1:6803/2999687971] [v2:127.0.0.1:6804/2999687971,v1:127.0.0.1:6805/2999687971] exists,up 477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:12:20.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:12:20.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:12:20.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:12:20.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:12:20.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:12:20.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:12:20.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:12:20.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:12:20.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:12:20.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:12:20.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:12:20.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:12:21.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:12:21.005 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:21.005 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:12:21.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:12:21.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:12:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-04-01T05:12:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-04-01T05:12:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-04-01T05:12:21.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:21.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:12:21.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-04-01T05:12:21.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-04-01T05:12:21.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672967' 2024-04-01T05:12:21.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:21.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:12:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-04-01T05:12:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-04-01T05:12:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672967 2-64424509445' 2024-04-01T05:12:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:21.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-04-01T05:12:21.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:21.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:12:21.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-04-01T05:12:21.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:21.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-04-01T05:12:21.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-04-01T05:12:21.323 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182402 2024-04-01T05:12:21.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:21.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 107374182402 2024-04-01T05:12:21.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:12:22.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:12:22.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:22.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-04-01T05:12:22.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:22.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-04-01T05:12:22.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:22.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:12:22.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-04-01T05:12:22.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:22.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-04-01T05:12:22.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-04-01T05:12:22.912 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672967 2024-04-01T05:12:22.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:12:23.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-04-01T05:12:23.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:23.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-04-01T05:12:23.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:23.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:12:23.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-04-01T05:12:23.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:23.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-04-01T05:12:23.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-04-01T05:12:23.197 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509445 2024-04-01T05:12:23.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:12:23.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-04-01T05:12:23.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:12:23.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:12:23.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:12:24.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:12:24.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:12:24.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:12:24.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:12:24.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:12:24.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:12:24.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:12:24.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:12:24.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:12:24.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:12:24.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:12:24.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:24.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:12:24.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-04-01T05:12:24.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-04-01T05:12:24.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:12:24.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:12:24.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:24.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:12:25.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-04-01T05:12:25.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-04-01T05:12:25.265 INFO:tasks.workunit.client.0.smithi067.stdout:Affected PG 1.0 w/ primary 1 2024-04-01T05:12:25.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-04-01T05:12:25.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:12:25.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:12:25.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:12:25.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:12:25.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-04-01T05:12:08.662054+0000 2024-04-01T05:12:25.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-04-01T05:12:25.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-04-01T05:12:25.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-04-01T05:12:25.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-04-01T05:12:25.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T05:12:25.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:12:25.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:12:25.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-04-01T05:12:08.662054+0000 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-04-01T05:12:25.864 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T05:12:25.865 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-01T05:12:25.865 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T05:12:25.865 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T05:12:25.865 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:12:25.866 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:12:25.866 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:12:25.866 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T05:12:25.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-04-01T05:12:25.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T05:12:25.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-04-01T05:12:26.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T05:12:26.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-04-01T05:12:26.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:12:26.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:12:26.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:12:26.323 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:26.323 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:12:26.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:12:26.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:26.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:12:26.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-04-01T05:12:26.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-04-01T05:12:26.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-04-01T05:12:26.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:26.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:12:26.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-04-01T05:12:26.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-04-01T05:12:26.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672969' 2024-04-01T05:12:26.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:26.557 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:12:26.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-04-01T05:12:26.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-04-01T05:12:26.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672969 2-64424509447' 2024-04-01T05:12:26.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:26.663 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-04-01T05:12:26.663 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:26.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:12:26.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-04-01T05:12:26.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:26.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-04-01T05:12:26.665 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 107374182404 2024-04-01T05:12:26.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-04-01T05:12:26.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:26.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182403 -lt 107374182404 2024-04-01T05:12:26.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:12:27.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:12:27.948 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:28.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-04-01T05:12:28.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:28.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-04-01T05:12:28.240 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:28.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:12:28.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-04-01T05:12:28.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:28.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-04-01T05:12:28.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-01T05:12:28.242 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672969 2024-04-01T05:12:28.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:12:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672969 2024-04-01T05:12:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:28.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-04-01T05:12:28.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:28.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:12:28.547 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-04-01T05:12:28.547 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:28.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-04-01T05:12:28.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-04-01T05:12:28.548 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509447 2024-04-01T05:12:28.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:12:28.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-04-01T05:12:28.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-04-01T05:12:28.831 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01 05:12:28.832086807+00:00 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T05:12:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-04-01T05:12:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-04-01T05:12:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:28.924 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:28.924 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:29.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:29.032 INFO:tasks.workunit.client.0.smithi067.stdout:{"deep":true,"must":false,"stamp":"2024-03-18T05:10:48.917512+0000"}0 ) state now: "active+clean" 2024-04-01T05:12:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-04-01T05:12:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:29.033 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:29.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-04-01T05:12:29.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:29.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:29.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:29.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:29.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:29.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:29.443 INFO:tasks.workunit.client.0.smithi067.stdout:1 ) state now: "active+clean" 2024-04-01T05:12:29.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-04-01T05:12:29.444 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-04-01T05:12:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:29.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:29.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:29.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:29.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:29.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:29.854 INFO:tasks.workunit.client.0.smithi067.stdout:2 ) state now: "active+clean" 2024-04-01T05:12:29.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-04-01T05:12:29.854 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:29.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:29.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:29.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-04-01T05:12:29.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:30.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:30.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:30.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:30.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:30.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:30.265 INFO:tasks.workunit.client.0.smithi067.stdout:3 ) state now: "active+clean" 2024-04-01T05:12:30.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-04-01T05:12:30.266 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:30.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:30.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:30.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-04-01T05:12:30.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:30.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:30.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:30.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:30.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:30.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:30.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-04-01T05:12:30.676 INFO:tasks.workunit.client.0.smithi067.stdout:4 ) state now: "active+clean" 2024-04-01T05:12:30.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:30.676 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:30.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:30.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-04-01T05:12:30.677 INFO:tasks.workunit.client.0.smithi067.stdout:loop --------> 4 2024-04-01T05:12:30.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-04-01T05:12:30.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:30.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:31.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:31.088 INFO:tasks.workunit.client.0.smithi067.stdout:5 ) state now: "active+clean" 2024-04-01T05:12:31.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-04-01T05:12:31.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:31.088 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:31.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:31.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-04-01T05:12:31.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:31.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:31.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:31.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:31.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:31.500 INFO:tasks.workunit.client.0.smithi067.stdout:6 ) state now: "active+clean" 2024-04-01T05:12:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-04-01T05:12:31.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:31.501 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-04-01T05:12:31.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:31.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:31.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:31.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:31.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:31.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:31.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2024-04-01T05:12:31.910 INFO:tasks.workunit.client.0.smithi067.stdout:7 ) state now: "active+clean" 2024-04-01T05:12:31.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:31.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:31.910 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:31.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 7 == 4 ']' 2024-04-01T05:12:31.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:32.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:32.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:32.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:32.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:32.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:32.321 INFO:tasks.workunit.client.0.smithi067.stdout:8 ) state now: "active+clean" 2024-04-01T05:12:32.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean"' 2024-04-01T05:12:32.322 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:32.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:32.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:32.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 8 == 4 ']' 2024-04-01T05:12:32.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:32.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:32.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:32.623 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:32.623 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:32.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:32.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 9 ') state now: ' '"active+clean"' 2024-04-01T05:12:32.732 INFO:tasks.workunit.client.0.smithi067.stdout:9 ) state now: "active+clean" 2024-04-01T05:12:32.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:32.733 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:32.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:32.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 9 == 4 ']' 2024-04-01T05:12:32.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:33.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:33.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:33.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:33.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:33.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:33.144 INFO:tasks.workunit.client.0.smithi067.stdout:10 ) state now: "active+clean" 2024-04-01T05:12:33.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 10 ') state now: ' '"active+clean"' 2024-04-01T05:12:33.144 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:33.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:33.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:33.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-04-01T05:12:33.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:33.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:33.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:33.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:33.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:33.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:33.554 INFO:tasks.workunit.client.0.smithi067.stdout:11 ) state now: "active+clean" 2024-04-01T05:12:33.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 11 ') state now: ' '"active+clean"' 2024-04-01T05:12:33.554 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:33.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:33.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:33.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-04-01T05:12:33.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:33.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:33.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 12 ') state now: ' '"active+clean"' 2024-04-01T05:12:33.963 INFO:tasks.workunit.client.0.smithi067.stdout:12 ) state now: "active+clean" 2024-04-01T05:12:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:33.964 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-04-01T05:12:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:34.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:34.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:34.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:34.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:34.374 INFO:tasks.workunit.client.0.smithi067.stdout:13 ) state now: "active+clean" 2024-04-01T05:12:34.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 13 ') state now: ' '"active+clean"' 2024-04-01T05:12:34.374 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:34.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:34.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:34.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-04-01T05:12:34.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:34.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:34.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:34.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:34.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:34.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:34.787 INFO:tasks.workunit.client.0.smithi067.stdout:14 ) state now: "active+clean" 2024-04-01T05:12:34.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 14 ') state now: ' '"active+clean"' 2024-04-01T05:12:34.787 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:34.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:34.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:34.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-04-01T05:12:34.788 INFO:tasks.workunit.client.0.smithi067.stdout:loop --------> 14 2024-04-01T05:12:34.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 14 2024-04-01T05:12:34.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:35.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:35.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:35.088 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:35.088 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:35.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 15 ') state now: ' '"active+clean"' 2024-04-01T05:12:35.198 INFO:tasks.workunit.client.0.smithi067.stdout:15 ) state now: "active+clean" 2024-04-01T05:12:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:35.199 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-04-01T05:12:35.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:35.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:35.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:35.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:35.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:35.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:35.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 16 ') state now: ' '"active+clean"' 2024-04-01T05:12:35.608 INFO:tasks.workunit.client.0.smithi067.stdout:16 ) state now: "active+clean" 2024-04-01T05:12:35.608 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:35.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:35.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:35.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-04-01T05:12:35.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:35.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:35.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:35.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:35.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 17 ') state now: ' '"active+clean"' 2024-04-01T05:12:36.018 INFO:tasks.workunit.client.0.smithi067.stdout:17 ) state now: "active+clean" 2024-04-01T05:12:36.018 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:36.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:36.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:36.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 7 == 4 ']' 2024-04-01T05:12:36.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:36.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:36.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:36.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:36.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:36.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:36.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 18 ') state now: ' '"active+clean"' 2024-04-01T05:12:36.428 INFO:tasks.workunit.client.0.smithi067.stdout:18 ) state now: "active+clean" 2024-04-01T05:12:36.429 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:36.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:36.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:36.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 8 == 4 ']' 2024-04-01T05:12:36.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:36.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:36.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:36.730 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:36.730 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:36.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:36.839 INFO:tasks.workunit.client.0.smithi067.stdout:19 ) state now: "active+clean" 2024-04-01T05:12:36.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 19 ') state now: ' '"active+clean"' 2024-04-01T05:12:36.839 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:36.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:36.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:36.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 9 == 4 ']' 2024-04-01T05:12:36.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:37.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:37.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:37.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:37.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:37.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:37.251 INFO:tasks.workunit.client.0.smithi067.stdout:20 ) state now: "active+clean" 2024-04-01T05:12:37.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 20 ') state now: ' '"active+clean"' 2024-04-01T05:12:37.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:37.251 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:37.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:37.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-04-01T05:12:37.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:37.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:37.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:37.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:37.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:37.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:37.663 INFO:tasks.workunit.client.0.smithi067.stdout:21 ) state now: "active+clean" 2024-04-01T05:12:37.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 21 ') state now: ' '"active+clean"' 2024-04-01T05:12:37.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:37.663 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:37.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:37.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-04-01T05:12:37.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:37.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:37.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:37.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:37.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:38.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:38.074 INFO:tasks.workunit.client.0.smithi067.stdout:22 ) state now: "active+clean" 2024-04-01T05:12:38.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 22 ') state now: ' '"active+clean"' 2024-04-01T05:12:38.075 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:38.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:38.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:38.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-04-01T05:12:38.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:38.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:38.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:38.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:38.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:38.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:38.484 INFO:tasks.workunit.client.0.smithi067.stdout:23 ) state now: "active+clean" 2024-04-01T05:12:38.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 23 ') state now: ' '"active+clean"' 2024-04-01T05:12:38.485 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:38.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:38.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:38.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-04-01T05:12:38.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:38.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:38.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:38.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:38.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:38.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:38.897 INFO:tasks.workunit.client.0.smithi067.stdout:24 ) state now: "active+clean" 2024-04-01T05:12:38.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 24 ') state now: ' '"active+clean"' 2024-04-01T05:12:38.898 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:38.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:38.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:38.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-04-01T05:12:38.899 INFO:tasks.workunit.client.0.smithi067.stdout:loop --------> 24 2024-04-01T05:12:38.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 24 2024-04-01T05:12:38.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:39.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:39.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:39.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:39.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:39.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:39.307 INFO:tasks.workunit.client.0.smithi067.stdout:25 ) state now: "active+clean" 2024-04-01T05:12:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 25 ') state now: ' '"active+clean"' 2024-04-01T05:12:39.308 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-04-01T05:12:39.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:39.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:39.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:39.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:39.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:39.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:39.718 INFO:tasks.workunit.client.0.smithi067.stdout:26 ) state now: "active+clean" 2024-04-01T05:12:39.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 26 ') state now: ' '"active+clean"' 2024-04-01T05:12:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:39.719 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-04-01T05:12:39.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:40.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:40.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:40.020 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:40.020 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:40.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:40.128 INFO:tasks.workunit.client.0.smithi067.stdout:27 ) state now: "active+clean" 2024-04-01T05:12:40.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 27 ') state now: ' '"active+clean"' 2024-04-01T05:12:40.128 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:40.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:40.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:40.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 7 == 4 ']' 2024-04-01T05:12:40.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:40.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:40.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:40.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:40.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:40.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-04-01T05:12:40.769 INFO:tasks.workunit.client.0.smithi067.stdout:28 ) state now: "active+clean" 2024-04-01T05:12:40.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 28 ') state now: ' '"active+clean"' 2024-04-01T05:12:40.770 INFO:tasks.workunit.client.0.smithi067.stdout:"active+clean" 2024-04-01T05:12:40.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:40.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-04-01T05:12:40.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 8 == 4 ']' 2024-04-01T05:12:40.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-04-01T05:12:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-04-01T05:12:41.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-04-01T05:12:41.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-04-01T05:12:41.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-04-01T05:12:41.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-04-01T05:12:41.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 29 ') state now: ' '"active+clean+scrubbing+deep"' 2024-04-01T05:12:41.182 INFO:tasks.workunit.client.0.smithi067.stdout:29 ) state now: "active+clean+scrubbing+deep" 2024-04-01T05:12:41.182 INFO:tasks.workunit.client.0.smithi067.stdout:found scrub 2024-04-01T05:12:41.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-04-01T05:12:41.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-04-01T05:12:41.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-04-01T05:12:41.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-04-01T05:12:41.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-04-01T05:12:41.184 INFO:tasks.workunit.client.0.smithi067.stdout:initiate_and_fetch_state ret: 0 2024-04-01T05:12:41.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-04-01T05:12:41.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T05:12:41.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.td/osd-scrub-repair.asok 2024-04-01T05:12:41.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-04-01T05:12:41.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-04-01T05:12:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-04-01T05:12:41.358 INFO:tasks.workunit.client.0.smithi067.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-04-01T05:12:41.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-04-01T05:12:41.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:41.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-04-01T05:12:41.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-04-01T05:12:41.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-04-01T05:12:41.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-04-01T05:12:41.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-04-01T05:12:41.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-04-01T05:12:41.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-04-01T05:12:41.768 INFO:tasks.workunit.client.0.smithi067.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-04-01T05:12:41.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-04-01T05:12:41.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:41.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-04-01T05:12:41.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-04-01T05:12:42.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-04-01T05:12:42.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-04-01T05:12:42.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-04-01T05:12:42.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-04-01T05:12:42.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-04-01T05:12:42.180 INFO:tasks.workunit.client.0.smithi067.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-04-01T05:12:42.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-04-01T05:12:42.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:42.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-04-01T05:12:42.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-04-01T05:12:42.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-04-01T05:12:42.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-04-01T05:12:42.482 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-04-01T05:12:42.482 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-04-01T05:12:42.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-04-01T05:12:42.591 INFO:tasks.workunit.client.0.smithi067.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-04-01T05:12:42.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-04-01T05:12:42.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:42.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-04-01T05:12:42.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-04-01T05:12:42.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-04-01T05:12:42.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-04-01T05:12:42.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-04-01T05:12:42.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-04-01T05:12:43.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-04-01T05:12:43.001 INFO:tasks.workunit.client.0.smithi067.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-04-01T05:12:43.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-04-01T05:12:43.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:43.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-04-01T05:12:43.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-04-01T05:12:43.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:12:43.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:12:43.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:12:43.284 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:43.284 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:12:43.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:12:43.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:43.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:12:43.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182408 2024-04-01T05:12:43.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182408 2024-04-01T05:12:43.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182408' 2024-04-01T05:12:43.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:43.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:12:43.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-04-01T05:12:43.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-04-01T05:12:43.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182408 1-42949672974' 2024-04-01T05:12:43.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:43.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:12:43.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-04-01T05:12:43.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-04-01T05:12:43.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182408 1-42949672974 2-64424509452' 2024-04-01T05:12:43.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:43.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182408 2024-04-01T05:12:43.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:43.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:12:43.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182408 2024-04-01T05:12:43.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:43.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182408 2024-04-01T05:12:43.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182408' 2024-04-01T05:12:43.605 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 107374182408 2024-04-01T05:12:43.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:43.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182408 -lt 107374182408 2024-04-01T05:12:43.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:43.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-04-01T05:12:43.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:43.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:12:43.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-04-01T05:12:43.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:43.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-04-01T05:12:43.891 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672974 2024-04-01T05:12:43.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-04-01T05:12:43.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:12:44.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-04-01T05:12:44.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:44.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-04-01T05:12:44.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:44.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:12:44.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-04-01T05:12:44.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:44.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-04-01T05:12:44.178 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509452 2024-04-01T05:12:44.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-04-01T05:12:44.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:12:44.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509452 2024-04-01T05:12:44.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:12:45.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:12:45.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:12:45.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-04-01T05:12:45.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-04-01T05:12:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-04-01T05:12:46.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-04-01T05:12:46.063 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-04-01T05:12:46.063 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-04-01T05:12:46.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-04-01T05:12:46.174 INFO:tasks.workunit.client.0.smithi067.stdout:wait-scrub-end state now: "active+clean" 2024-04-01T05:12:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-04-01T05:12:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-04-01T05:12:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-04-01T05:12:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-04-01T05:12:46.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-04-01T05:12:46.439 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T05:12:46.439 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-04-01T05:12:41.093496+0000 21'1 26:43 [1,0] 1 [1,0] 1 0'0 2024-03-18T05:10:48.917512+0000 0'0 2024-03-18T05:10:48.917512+0000 0 0 deep scrubbing for 0s 0 0 2024-04-01T05:12:46.439 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T05:12:46.439 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T05:12:46.439 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:46.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:12:46.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:12:46.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:46.761 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:12:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:12:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:12:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:12:47.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:12:47.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:12:47.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:12:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-04-01T05:12:48.530 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:12:48.530 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:12:48.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:12:48.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:12:48.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:12:48.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:12:48.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:12:48.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:12:48.799 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:12:48.800 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:12:48.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:12:48.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:12:48.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:12:48.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:12:48.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:12:48.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:12:48.833 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:48.831+0000 7fad05fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:48.834 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:48.834+0000 7fad05fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:48.835 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:48.835+0000 7fad05fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:12:49.093 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:12:49.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:49.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:49.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:49.355+0000 7fad05fa8640 -1 Falling back to public interface 2024-04-01T05:12:49.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:50.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:50.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:50.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:12:50.374 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:50.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:50.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:50.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:51.581 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:12:51.578+0000 7fad05fa8640 -1 osd.0 26 log_to_monitors true 2024-04-01T05:12:51.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:51.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:51.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:12:51.695 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:12:51.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:51.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:52.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:12:53.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:12:53.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:12:53.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:12:53.024 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:12:53.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:12:53.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/477128131,v1:127.0.0.1:6803/477128131] [v2:127.0.0.1:6804/477128131,v1:127.0.0.1:6805/477128131] exists,up 477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:12:53.321 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:12:53.322 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:12:53.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:12:53.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:12:53.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:12:53.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:12:53.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:12:53.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:12:53.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:12:53.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:12:53.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:12:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:12:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:12:53.708 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:12:53.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:12:53.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:53.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:12:53.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-04-01T05:12:53.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-04-01T05:12:53.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-04-01T05:12:53.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:53.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:12:53.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-04-01T05:12:53.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-04-01T05:12:53.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672977' 2024-04-01T05:12:53.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:12:53.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:12:54.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672977 2-64424509455' 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 128849018882 2024-04-01T05:12:54.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:54.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-04-01T05:12:54.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:12:55.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:12:55.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:55.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-04-01T05:12:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:12:56.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:12:56.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:12:56.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-04-01T05:12:56.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:56.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-04-01T05:12:56.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:56.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:12:56.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-04-01T05:12:56.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:56.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-04-01T05:12:56.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-01T05:12:56.937 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672977 2024-04-01T05:12:56.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:12:57.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-01T05:12:57.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:12:57.263 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-04-01T05:12:57.263 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:12:57.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:12:57.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-04-01T05:12:57.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:12:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-04-01T05:12:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-01T05:12:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509455 2024-04-01T05:12:57.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:12:57.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-01T05:12:57.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:12:57.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:12:57.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:12:57.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:12:58.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:12:58.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:12:58.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:12:58.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:12:58.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:12:58.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:12:58.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:12:58.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-04-01T05:12:58.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T05:12:58.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:12:58.588 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-04-01T05:12:58.589 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T05:12:58.589 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:12:58.589 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:58.589 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:12:58.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:12:58.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-04-01T05:12:58.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:12:59.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:12:59.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:12:59.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:12:59.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:12:59.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-04-01T05:13:01.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:13:01.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:13:01.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:13:01.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:13:01.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:13:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:13:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:13:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:13:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:13:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:13:01.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:13:01.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:13:01.023 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:13:01.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-04-01T05:13:01.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:13:01.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:13:01.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:13:01.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:13:01.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:13:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:01.051+0000 7f28fcc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:01.054 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:01.054+0000 7f28fcc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:01.055+0000 7f28fcc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:01.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:01.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:01.570+0000 7f28fcc15640 -1 Falling back to public interface 2024-04-01T05:13:01.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:02.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:02.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:02.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:13:02.599 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:13:02.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:02.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:02.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:03.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:03.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:03.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:13:03.894 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:13:03.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:03.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:03.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:03.979+0000 7f28fcc15640 -1 osd.0 31 log_to_monitors true 2024-04-01T05:13:04.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:13:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:13:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:05.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/496905364,v1:127.0.0.1:6803/496905364] [v2:127.0.0.1:6804/496905364,v1:127.0.0.1:6805/496905364] exists,up 477b6684-b468-45fb-a574-a334ef8353ed 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:13:05.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:13:05.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:13:05.588 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:13:05.588 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:13:05.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:13:05.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:13:05.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:13:05.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:13:05.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:13:05.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:13:05.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:13:05.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:13:05.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:13:05.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:13:05.985 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:13:05.985 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:13:05.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:13:05.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:13:05.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:13:06.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-04-01T05:13:06.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-04-01T05:13:06.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-04-01T05:13:06.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:13:06.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:13:06.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-04-01T05:13:06.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-04-01T05:13:06.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672980' 2024-04-01T05:13:06.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:13:06.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:13:06.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-04-01T05:13:06.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-04-01T05:13:06.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672980 2-64424509458' 2024-04-01T05:13:06.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:13:06.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-04-01T05:13:06.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:13:06.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:13:06.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-04-01T05:13:06.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:13:06.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-04-01T05:13:06.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-04-01T05:13:06.303 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 150323855362 2024-04-01T05:13:06.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:13:06.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-04-01T05:13:06.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:13:07.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:13:07.589 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:13:07.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-04-01T05:13:07.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:13:07.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-04-01T05:13:07.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:13:07.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:13:07.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-04-01T05:13:07.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:13:07.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-04-01T05:13:07.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-04-01T05:13:07.905 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672980 2024-04-01T05:13:07.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:13:08.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-04-01T05:13:08.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:13:08.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-04-01T05:13:08.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:13:08.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:13:08.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-04-01T05:13:08.235 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:13:08.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-04-01T05:13:08.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-01T05:13:08.236 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509458 2024-04-01T05:13:08.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:13:08.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-01T05:13:08.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:13:08.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:13:08.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:13:08.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T05:13:08.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:13:08.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:13:08.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:13:08.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:13:08.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:13:08.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:13:08.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:13:09.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T05:13:09.204 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:13:09.204 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:13:09.204 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:13:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T05:13:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:13:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:13:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:13:09.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-04-01T05:13:09.576 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:12:44.104+0000 7f05ffa94640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-04-01T05:13:09.576 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:12:44.104+0000 7f05ffa94640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-04-01T05:13:09.576 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:12:44.104+0000 7f05ffa94640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-04-01T05:13:09.577 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:12:44.104+0000 7f05ffa94640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-04-01T05:13:09.577 INFO:tasks.workunit.client.0.smithi067.stdout:2024-04-01T05:12:44.104+0000 7f05ffa94640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:13:09.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:13:09.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:13:09.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:13:09.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:13:09.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:13:09.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:13:09.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:13:09.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:13:09.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:13:09.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:13:09.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:13:09.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:13:09.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:13:09.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:13:09.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:13:09.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:13:09.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:09.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:09.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:13:09.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:13:09.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:13:09.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:13:09.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:13:09.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:13:09.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:13:09.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:13:09.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:13:09.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:13:09.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:13:09.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:13:09.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:13:09.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:13:09.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:13:09.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:13:09.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:13:09.711 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:09.711 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:09.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:13:09.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:13:09.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:13:09.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:13:09.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:13:09.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:09.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:09.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:13:09.714 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:13:09.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:13:09.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:13:09.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:10.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:10.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:13:10.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:13:10.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:13:10.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:13:10.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:13:10.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:13:10.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:13:10.395 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:13:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:13:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:10.396 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:13:10.396 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:13:10.396 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:13:10.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-04-01T05:13:10.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:13:10.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:13:10.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:13:10.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:13:10.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:13:10.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:13:11.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:11.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:13:11.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:13:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:13:11.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:13:11.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:11.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:13:11.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:13:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58298ff1-d2f7-4057-b194-7058b4abb4c0 2024-04-01T05:13:11.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 58298ff1-d2f7-4057-b194-7058b4abb4c0' 2024-04-01T05:13:11.307 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 58298ff1-d2f7-4057-b194-7058b4abb4c0 2024-04-01T05:13:11.307 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:13:11.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnQgpmqi0qExAATn/XeuV9pNIj5BUmMB/V/Q== 2024-04-01T05:13:11.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnQgpmqi0qExAATn/XeuV9pNIj5BUmMB/V/Q=="}' 2024-04-01T05:13:11.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58298ff1-d2f7-4057-b194-7058b4abb4c0 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:13:11.569 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:13:11.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:13:11.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBnQgpmqi0qExAATn/XeuV9pNIj5BUmMB/V/Q== --osd-uuid 58298ff1-d2f7-4057-b194-7058b4abb4c0 2024-04-01T05:13:11.612 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.611+0000 7fcc71811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:11.620 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.619+0000 7fcc71811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:11.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.621+0000 7fcc71811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:11.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.621+0000 7fcc71811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:13:11.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.621+0000 7fcc71811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:13:11.622 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.621+0000 7fcc71811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:13:11.712 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:11.711+0000 7fcc71811640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:13:17.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:13:17.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:13:17.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:13:17.186 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:13:17.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:13:17.578 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:13:17.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:13:17.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:13:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:13:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:13:17.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:13:17.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:17.608+0000 7fbb3c297640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:17.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:17.636+0000 7fbb3c297640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:17.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:17.637+0000 7fbb3c297640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:13:17.861 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:13:17.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:17.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:18.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:18.155 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:18.154+0000 7fbb3c297640 -1 Falling back to public interface 2024-04-01T05:13:19.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:19.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:19.142 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:13:19.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:13:19.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:19.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:19.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:20.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:13:20.435 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:13:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:20.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:20.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:20.443+0000 7fbb3c297640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:13:20.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:21.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:21.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:21.888 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:13:21.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:13:21.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:21.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:23.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:23.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:23.235 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:13:23.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:13:23.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:23.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:23.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:24.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:13:24.530 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:13:24.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:24.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:24.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:25.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:25.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:25.824 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:13:25.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:13:25.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:25.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:26.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:27.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:27.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:27.185 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:13:27.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:13:27.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:27.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:27.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:28.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:28.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:28.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:13:28.481 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:13:28.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:28.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:28.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:29.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:29.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:29.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:13:29.777 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:13:29.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:29.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:30.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:31.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:31.186 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:13:31.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:31.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:13:31.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:31.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:31.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:32.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:32.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:32.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:13:32.767 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:13:32.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:32.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:33.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:34.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:34.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:34.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:13:34.228 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:13:34.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:34.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:34.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:35.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:35.806 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:13:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:13:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:36.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:37.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:37.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:37.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:13:37.339 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:13:37.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:37.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:37.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:38.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:38.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:38.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:13:38.990 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:13:38.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:38.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:39.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:40.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:40.617 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:13:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:13:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:40.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:41.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:42.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:42.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:42.056 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:13:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:13:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:13:42.349 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2377070607,v1:127.0.0.1:6803/2377070607] [v2:127.0.0.1:6804/2377070607,v1:127.0.0.1:6805/2377070607] exists,up 58298ff1-d2f7-4057-b194-7058b4abb4c0 2024-04-01T05:13:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:13:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:13:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:13:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:13:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:13:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:13:42.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a581aab5-0545-4bb0-b774-ed8b94f31215 2024-04-01T05:13:42.357 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 a581aab5-0545-4bb0-b774-ed8b94f31215 2024-04-01T05:13:42.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a581aab5-0545-4bb0-b774-ed8b94f31215' 2024-04-01T05:13:42.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:13:42.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGQgpmGvQZFhAA43Gm89eMyYwQnxKc1bjbug== 2024-04-01T05:13:42.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGQgpmGvQZFhAA43Gm89eMyYwQnxKc1bjbug=="}' 2024-04-01T05:13:42.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a581aab5-0545-4bb0-b774-ed8b94f31215 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:13:42.705 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:13:42.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:13:42.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCGQgpmGvQZFhAA43Gm89eMyYwQnxKc1bjbug== --osd-uuid a581aab5-0545-4bb0-b774-ed8b94f31215 2024-04-01T05:13:42.748 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.748+0000 7f884160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:42.781 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.781+0000 7f884160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:42.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.782+0000 7f884160f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:42.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.783+0000 7f884160f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:13:42.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.783+0000 7f884160f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:13:42.783 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.783+0000 7f884160f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:13:42.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:42.857+0000 7f884160f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:13:48.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:13:48.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:13:48.759 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:13:48.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:13:48.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:13:49.129 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:13:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:13:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:13:49.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:13:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:13:49.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:13:49.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:49.159+0000 7f7fed3aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:49.161 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:49.161+0000 7f7fed3aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:49.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:49.162+0000 7f7fed3aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:13:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:13:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:13:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:13:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:13:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:13:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:13:49.417 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:13:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:49.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:49.677+0000 7f7fed3aa640 -1 Falling back to public interface 2024-04-01T05:13:49.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:50.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:50.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:50.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:13:50.717 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:13:50.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:50.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:51.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:52.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:52.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:52.014 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:13:52.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:13:52.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:52.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:52.061 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:13:52.061+0000 7f7fed3aa640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:13:52.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:53.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:53.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:53.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:13:53.356 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:13:53.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:53.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:53.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:54.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:54.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:54.828 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:13:54.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:13:54.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:54.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:55.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:56.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:56.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:56.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:13:56.119 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:13:56.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:56.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:56.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:57.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:57.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:57.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:13:57.491 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:13:57.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:57.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:57.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:13:58.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:13:58.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:13:58.887 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:13:58.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:13:58.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:13:58.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:13:59.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:00.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:00.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:00.244 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:14:00.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:14:00.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:00.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:00.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:01.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:01.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:01.607 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:14:01.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:14:01.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:01.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:01.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:02.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:02.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:02.954 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:14:02.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:14:02.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:02.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:03.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:04.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:14:04.278 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:14:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:04.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:05.945 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:14:05.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:14:05.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:05.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:06.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:07.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:07.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:14:07.267 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:14:07.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:07.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:08.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:09.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:09.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:09.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:14:09.174 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:14:09.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:09.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:09.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:10.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:10.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:10.723 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:14:10.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:14:10.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:10.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:11.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:12.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:12.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:12.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:14:12.359 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:14:12.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:12.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:12.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:13.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:13.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:13.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:14:13.976 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:14:13.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:13.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:14.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:15.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:15.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:15.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:14:15.275 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:14:15.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:15.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:14:15.564 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4116234818,v1:127.0.0.1:6811/4116234818] [v2:127.0.0.1:6812/4116234818,v1:127.0.0.1:6813/4116234818] exists,up a581aab5-0545-4bb0-b774-ed8b94f31215 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:14:15.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:14:15.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=185b6f47-a3ca-4851-aa93-8ed5a037df46 2024-04-01T05:14:15.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 185b6f47-a3ca-4851-aa93-8ed5a037df46' 2024-04-01T05:14:15.571 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 185b6f47-a3ca-4851-aa93-8ed5a037df46 2024-04-01T05:14:15.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:14:15.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnQgpmICfdIhAA3xIGiBZUQ2k0Bb3vGtEZVg== 2024-04-01T05:14:15.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnQgpmICfdIhAA3xIGiBZUQ2k0Bb3vGtEZVg=="}' 2024-04-01T05:14:15.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 185b6f47-a3ca-4851-aa93-8ed5a037df46 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:14:15.951 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:14:15.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:14:15.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCnQgpmICfdIhAA3xIGiBZUQ2k0Bb3vGtEZVg== --osd-uuid 185b6f47-a3ca-4851-aa93-8ed5a037df46 2024-04-01T05:14:15.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:15.994+0000 7f2ff180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:14:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:16.036+0000 7f2ff180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:14:16.036 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:16.037+0000 7f2ff180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:14:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:16.037+0000 7f2ff180d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:14:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:16.037+0000 7f2ff180d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:14:16.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:16.037+0000 7f2ff180d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:14:16.110 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:16.110+0000 7f2ff180d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:14:22.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:14:22.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:14:22.339 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:14:22.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:14:22.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:14:22.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:14:22.723 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:14:22.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:14:22.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:14:22.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:14:22.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:14:22.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:22.752+0000 7f4a67aa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:14:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:22.754+0000 7f4a67aa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:14:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:22.755+0000 7f4a67aa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:14:23.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:14:23.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:14:23.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:14:23.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:14:23.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:14:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:14:23.045 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:14:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:23.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:23.274+0000 7f4a67aa4640 -1 Falling back to public interface 2024-04-01T05:14:23.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:24.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:24.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:24.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:14:24.372 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:14:24.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:24.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:24.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:25.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:25.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:25.691 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:14:25.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:14:25.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:25.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:14:25.704+0000 7f4a67aa4640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:14:26.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:27.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:27.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:27.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:14:27.013 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:14:27.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:27.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:27.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:28.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:28.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:28.315 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:14:28.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:14:28.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:28.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:28.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:29.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:29.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:29.713 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:14:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:14:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:31.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:31.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:31.039 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:14:31.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:14:31.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:31.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:31.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:32.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:32.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:32.344 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:14:32.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:14:32.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:32.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:32.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:33.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:33.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:33.814 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:14:33.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:14:33.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:33.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:34.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:35.108 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:14:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:14:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:35.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:36.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:36.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:36.507 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:14:36.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:14:36.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:36.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:37.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:38.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:38.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:38.072 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:14:38.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:14:38.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:38.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:38.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:39.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:39.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:39.368 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:14:39.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:14:39.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:39.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:39.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:40.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:14:40.730 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:14:40.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:40.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:41.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:42.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:42.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:42.381 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:14:42.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:14:42.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:42.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:42.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:43.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:43.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:43.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:14:43.986 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:14:43.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:43.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:44.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:45.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:45.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:45.886 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:14:45.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:14:45.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:45.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:46.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:47.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:47.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:47.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:14:47.698 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:14:47.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:47.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:48.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:14:49.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:14:49.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:14:49.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:14:49.315 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:14:49.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:14:49.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:14:49.606 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3170400727,v1:127.0.0.1:6819/3170400727] [v2:127.0.0.1:6820/3170400727,v1:127.0.0.1:6821/3170400727] exists,up 185b6f47-a3ca-4851-aa93-8ed5a037df46 2024-04-01T05:14:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:14:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:14:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:14:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-04-01T05:14:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:14:49.881 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:14:49.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:14:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:14:50.473 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:14:50.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:14:51.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:14:53.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-04-01T05:14:53.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:14:53.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:14:53.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:14:53.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:14:53.698 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:14:53.698 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:14:53.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:14:53.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:14:53.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:14:53.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:14:53.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:14:53.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:14:53.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:14:53.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:14:53.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:14:54.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:14:54.118 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:14:54.118 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:14:54.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:14:54.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:14:54.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:14:54.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-04-01T05:14:54.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-04-01T05:14:54.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-04-01T05:14:54.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:14:54.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:14:54.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476749 2024-04-01T05:14:54.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476749 2024-04-01T05:14:54.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-68719476749' 2024-04-01T05:14:54.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:14:54.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:14:54.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953481 2024-04-01T05:14:54.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953481 2024-04-01T05:14:54.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-68719476749 2-137438953481' 2024-04-01T05:14:54.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:14:54.493 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-04-01T05:14:54.493 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:14:54.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:14:54.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-04-01T05:14:54.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:14:54.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-04-01T05:14:54.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-01T05:14:54.496 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836498 2024-04-01T05:14:54.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:14:54.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-04-01T05:14:54.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:14:55.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:14:55.802 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:14:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-01T05:14:56.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:14:56.097 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476749 2024-04-01T05:14:56.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:14:56.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:14:56.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476749 2024-04-01T05:14:56.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:14:56.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476749 2024-04-01T05:14:56.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476749' 2024-04-01T05:14:56.101 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 68719476749 2024-04-01T05:14:56.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:14:56.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476749 -lt 68719476749 2024-04-01T05:14:56.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:14:56.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953481 2024-04-01T05:14:56.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:14:56.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:14:56.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953481 2024-04-01T05:14:56.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:14:56.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953481 2024-04-01T05:14:56.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953481' 2024-04-01T05:14:56.470 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 137438953481 2024-04-01T05:14:56.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:14:56.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953481 -lt 137438953481 2024-04-01T05:14:56.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:14:56.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:14:56.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:14:57.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:14:57.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:14:57.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:14:57.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:14:57.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:14:57.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:14:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:14:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:14:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:14:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-04-01T05:14:57.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:14:57.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:14:57.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-04-01T05:14:57.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:14:57.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-01T05:14:58.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:14:58.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:14:59.211 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:14:59.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:15:00.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-04-01T05:15:00.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:15:00.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:15:00.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:15:00.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:15:00.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:15:00.226 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:15:00.226 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:15:00.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:15:00.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:15:00.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:15:00.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:15:00.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:15:00.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:15:00.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:15:00.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:15:00.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:15:00.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:15:00.624 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:15:00.624 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:15:00.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:15:00.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:00.624 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:15:00.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-04-01T05:15:00.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-04-01T05:15:00.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-04-01T05:15:00.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:00.729 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:15:00.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476752 2024-04-01T05:15:00.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476752 2024-04-01T05:15:00.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-68719476752' 2024-04-01T05:15:00.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:00.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:15:00.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953483 2024-04-01T05:15:00.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953483 2024-04-01T05:15:00.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-68719476752 2-137438953483' 2024-04-01T05:15:00.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:00.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-04-01T05:15:00.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:00.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:15:00.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-04-01T05:15:00.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-04-01T05:15:00.941 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836500 2024-04-01T05:15:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-01T05:15:00.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:01.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-04-01T05:15:01.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:15:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:15:02.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:02.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-04-01T05:15:02.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:02.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476752 2024-04-01T05:15:02.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:15:02.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476752 2024-04-01T05:15:02.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476752 2024-04-01T05:15:02.520 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 68719476752 2024-04-01T05:15:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476752' 2024-04-01T05:15:02.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:15:02.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476752 -lt 68719476752 2024-04-01T05:15:02.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:02.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953483 2024-04-01T05:15:02.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:02.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:15:02.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953483 2024-04-01T05:15:02.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:02.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953483 2024-04-01T05:15:02.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953483' 2024-04-01T05:15:02.825 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 137438953483 2024-04-01T05:15:02.825 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:15:03.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953483 -lt 137438953483 2024-04-01T05:15:03.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:15:03.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:03.157 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:03.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:15:03.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:15:03.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:15:03.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:15:03.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:03.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-04-01T05:15:04.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:15:04.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-04-01T05:15:04.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-04-01T05:15:04.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:15:04.409 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-04-01T05:15:04.409 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:15:04.409 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:15:04.409 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:15:04.409 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:15:04.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-04-01T05:15:04.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:15:04.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:15:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:15:05.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:15:05.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:15:05.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:15:05.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:15:05.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:15:05.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:15:05.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:15:05.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:15:05.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:15:06.966 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:15:06.966 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-04-01T05:15:06.966 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:15:07.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:15:07.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:15:07.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:15:07.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:15:07.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:15:07.489 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:15:07.489 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:15:07.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:15:07.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:15:07.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:15:07.491 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:15:07.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:15:07.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:15:07.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:15:07.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:15:07.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:15:07.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:15:07.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:07.521+0000 7f91f9ac6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:07.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:07.523+0000 7f91f9ac6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:07.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:07.524+0000 7f91f9ac6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:07.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:07.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:08.040 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:08.040+0000 7f91f9ac6640 -1 Falling back to public interface 2024-04-01T05:15:08.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:09.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:09.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:09.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:15:09.097 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:15:09.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:09.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:09.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:15:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:15:10.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:10.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:10.415 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:10.415+0000 7f91f9ac6640 -1 osd.1 53 log_to_monitors true 2024-04-01T05:15:10.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:11.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:11.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:11.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:15:11.848 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:15:11.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:11.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:12.237 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 55 last_clean_interval [16,53) [v2:127.0.0.1:6810/1628090950,v1:127.0.0.1:6811/1628090950] [v2:127.0.0.1:6812/1628090950,v1:127.0.0.1:6813/1628090950] exists,up a581aab5-0545-4bb0-b774-ed8b94f31215 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:15:12.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:15:12.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:15:12.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:15:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:15:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:15:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:15:12.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:15:12.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:15:12.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:15:12.664 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:15:12.665 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:15:12.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:15:12.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:12.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:15:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-04-01T05:15:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-04-01T05:15:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-04-01T05:15:12.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:12.798 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:15:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-04-01T05:15:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-04-01T05:15:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-257698037762' 2024-04-01T05:15:12.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:12.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:15:13.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953486 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953486 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-257698037762 2-137438953486' 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-04-01T05:15:13.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:13.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-04-01T05:15:13.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-01T05:15:13.012 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836504 2024-04-01T05:15:13.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:13.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-04-01T05:15:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:15:14.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:15:14.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:14.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-01T05:15:14.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:14.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037762 2024-04-01T05:15:14.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:14.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:15:14.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037762 2024-04-01T05:15:14.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-04-01T05:15:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2024-04-01T05:15:14.619 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 257698037762 2024-04-01T05:15:14.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:15:14.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-04-01T05:15:14.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:14.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953486 2024-04-01T05:15:14.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:14.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:15:14.938 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953486 2024-04-01T05:15:14.938 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953486 2024-04-01T05:15:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953486' 2024-04-01T05:15:14.939 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 137438953486 2024-04-01T05:15:14.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:15:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953486 -lt 137438953486 2024-04-01T05:15:15.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:15:15.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:15.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:15:15.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:15:15.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:15:15.865 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:15:15.865 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:15.865 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:15:16.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:15:16.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-04-01T05:15:16.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-04-01T05:15:16.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:15:16.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:15:16.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:15:16.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:15:16.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-04-01T05:15:04.927794+0000 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:15:04.927794+0000 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:15:16.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:15:17.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:15:04.927794+0000 '>' 2024-04-01T05:15:04.927794+0000 2024-04-01T05:15:17.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:15:18.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:15:18.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:15:18.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:15:18.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:15:18.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:15:18.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:15:18.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:15:18.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:15:04.927794+0000 '>' 2024-04-01T05:15:04.927794+0000 2024-04-01T05:15:18.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:15:19.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:15:19.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:15:19.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:15:19.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:15:19.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:15:19.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:15:19.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:15:19.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:15:15.870559+0000 '>' 2024-04-01T05:15:04.927794+0000 2024-04-01T05:15:19.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:15:19.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-04-01T05:15:19.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:15:19.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:15:19.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:15:19.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:15:19.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:15:19.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:15:19.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:15:19.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:15:19.827 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:15:20.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:15:20.107 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:15:20.107 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:15:20.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:15:20.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:20.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:15:20.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-04-01T05:15:20.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-04-01T05:15:20.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-04-01T05:15:20.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:20.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:15:20.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037764 2024-04-01T05:15:20.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037764 2024-04-01T05:15:20.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-257698037764' 2024-04-01T05:15:20.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:20.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:15:20.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953489 2024-04-01T05:15:20.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953489 2024-04-01T05:15:20.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-257698037764 2-137438953489' 2024-04-01T05:15:20.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:20.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-04-01T05:15:20.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:20.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:15:20.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-04-01T05:15:20.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-04-01T05:15:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-04-01T05:15:20.428 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836506 2024-04-01T05:15:20.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-04-01T05:15:20.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:15:21.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:15:21.711 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:22.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-04-01T05:15:22.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:22.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037764 2024-04-01T05:15:22.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:22.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:15:22.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037764 2024-04-01T05:15:22.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:22.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037764 2024-04-01T05:15:22.008 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 257698037764 2024-04-01T05:15:22.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037764' 2024-04-01T05:15:22.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:15:22.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037765 -lt 257698037764 2024-04-01T05:15:22.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:22.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953489 2024-04-01T05:15:22.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:22.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:15:22.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953489 2024-04-01T05:15:22.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:22.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953489 2024-04-01T05:15:22.297 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 137438953489 2024-04-01T05:15:22.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953489' 2024-04-01T05:15:22.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:15:22.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953489 -lt 137438953489 2024-04-01T05:15:22.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:15:22.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:22.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:22.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:15:22.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:15:22.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:15:22.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:15:22.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:15:22.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:15:22.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:15:22.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:15:23.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:15:23.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:15:23.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:23.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:15:23.614 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:15:23.615 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:15:23.615 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:15:23.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-04-01T05:15:23.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:15:23.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:15:24.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:15:24.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:15:24.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:15:24.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:15:24.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:15:24.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:15:24.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:15:26.206 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:15:26.206 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-04-01T05:15:26.207 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:15:26.207 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:15:26.207 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:15:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:26.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:15:26.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:15:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:15:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:15:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:15:26.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:15:26.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:26.518+0000 7ff5a46aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:26.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:26.520+0000 7ff5a46aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:26.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:26.521+0000 7ff5a46aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:26.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:27.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:27.036+0000 7ff5a46aa640 -1 Falling back to public interface 2024-04-01T05:15:27.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:28.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:28.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:28.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:15:28.062 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:15:28.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:28.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:28.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:29.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:29.337+0000 7ff5a46aa640 -1 osd.1 68 log_to_monitors true 2024-04-01T05:15:29.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:29.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:29.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:15:29.354 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:15:29.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:29.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:29.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:30.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:30.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:15:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:15:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 69 last_clean_interval [60,68) [v2:127.0.0.1:6810/2957751765,v1:127.0.0.1:6811/2957751765] [v2:127.0.0.1:6812/2957751765,v1:127.0.0.1:6813/2957751765] exists,up a581aab5-0545-4bb0-b774-ed8b94f31215 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:15:31.113 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:15:31.114 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:15:31.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:15:31.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:15:31.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:15:31.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:15:31.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:15:31.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:15:31.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:15:31.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:15:31.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:15:31.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:15:31.597 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:15:31.597 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:15:31.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:15:31.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:31.597 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:15:31.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-04-01T05:15:31.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-04-01T05:15:31.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-04-01T05:15:31.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:31.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:15:31.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-04-01T05:15:31.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-04-01T05:15:31.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-322122547202' 2024-04-01T05:15:31.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:15:31.829 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:15:31.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953492 2024-04-01T05:15:31.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953492 2024-04-01T05:15:31.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-322122547202 2-137438953492' 2024-04-01T05:15:31.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:31.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-04-01T05:15:31.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:31.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:15:31.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-04-01T05:15:31.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-04-01T05:15:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-01T05:15:31.936 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836509 2024-04-01T05:15:31.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:32.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-04-01T05:15:32.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:15:33.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:15:33.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:33.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-04-01T05:15:33.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:15:34.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:15:34.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:15:34.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836509 2024-04-01T05:15:34.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:34.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547202 2024-04-01T05:15:34.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:34.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:15:34.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547202 2024-04-01T05:15:34.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:34.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-04-01T05:15:34.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-04-01T05:15:34.920 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 322122547202 2024-04-01T05:15:34.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:15:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547202 -lt 322122547202 2024-04-01T05:15:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:15:35.205 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953492 2024-04-01T05:15:35.205 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:15:35.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:15:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953492 2024-04-01T05:15:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:15:35.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953492 2024-04-01T05:15:35.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953492' 2024-04-01T05:15:35.207 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 137438953492 2024-04-01T05:15:35.208 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:15:35.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953492 -lt 137438953492 2024-04-01T05:15:35.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:15:35.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:35.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:15:35.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:15:36.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:15:36.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:15:36.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:15:36.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:15:36.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:15:36.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:15:36.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:15:36.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:15:36.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:15:36.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:15:36.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:15:36.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:15:36.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:15:36.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:15:36.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:15:36.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:15:36.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:15:36.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:15:36.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:15:36.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:15:36.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:15:36.677 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:15:36.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:15:36.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:15:36.678 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:15:36.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:15:36.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:15:36.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:15:36.680 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:15:36.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:15:36.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:15:36.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:15:36.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:15:36.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:36.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:36.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:15:36.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:15:36.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:15:36.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:15:36.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:15:36.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:15:36.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:15:36.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:15:36.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:15:36.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:15:36.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:15:36.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:15:36.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:15:36.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:15:36.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:15:36.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:15:36.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:36.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:36.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:15:36.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:15:36.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:15:36.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:15:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:15:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:36.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-04-01T05:15:36.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:15:36.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:37.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:15:37.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:15:37.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:15:37.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:15:37.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:15:37.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:15:37.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:15:37.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:15:37.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-04-01T05:15:37.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:15:37.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:15:37.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:15:37.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:15:37.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:15:37.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:15:38.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:38.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:15:38.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:15:38.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:15:38.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-04-01T05:15:38.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-04-01T05:15:38.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:15:38.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:15:38.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:15:38.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:15:38.211 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:15:38.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:15:38.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:15:38.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:15:38.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63b3b5b3-cdd0-46fc-9312-d189f0118f60 2024-04-01T05:15:38.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 63b3b5b3-cdd0-46fc-9312-d189f0118f60' 2024-04-01T05:15:38.213 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 63b3b5b3-cdd0-46fc-9312-d189f0118f60 2024-04-01T05:15:38.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:15:38.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6QgpmIrujDRAA5a7CM8/MfWF/C7bzv7mOFQ== 2024-04-01T05:15:38.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6QgpmIrujDRAA5a7CM8/MfWF/C7bzv7mOFQ=="}' 2024-04-01T05:15:38.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63b3b5b3-cdd0-46fc-9312-d189f0118f60 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:15:38.433 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:15:38.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:15:38.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD6QgpmIrujDRAA5a7CM8/MfWF/C7bzv7mOFQ== --osd-uuid 63b3b5b3-cdd0-46fc-9312-d189f0118f60 2024-04-01T05:15:38.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.475+0000 7fc40bffd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:38.484 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.484+0000 7fc40bffd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:38.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.485+0000 7fc40bffd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:38.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.485+0000 7fc40bffd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:15:38.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.485+0000 7fc40bffd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:15:38.485 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.485+0000 7fc40bffd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:15:38.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:38.567+0000 7fc40bffd640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:15:44.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:15:44.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:15:44.083 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:15:44.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:15:44.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:15:44.453 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:15:44.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:15:44.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:15:44.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:15:44.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:15:44.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:15:44.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:44.483+0000 7fd146a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:44.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:44.524+0000 7fd146a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:44.525 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:44.525+0000 7fd146a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:15:44.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:15:44.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:15:44.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:15:44.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:15:44.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:15:44.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:44.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:15:44.742 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:15:44.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:44.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:45.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:45.050+0000 7fd146a13640 -1 Falling back to public interface 2024-04-01T05:15:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:46.028 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:15:46.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:15:46.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:46.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:47.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:47.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:47.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:15:47.316 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:15:47.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:47.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:47.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:15:47.372+0000 7fd146a13640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:15:47.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:48.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:48.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:48.649 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:15:48.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:15:48.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:48.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:48.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:49.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:49.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:49.947 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:15:49.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:15:49.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:49.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:50.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:51.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:51.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:51.341 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:15:51.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:15:51.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:51.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:51.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:52.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:52.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:52.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:15:52.634 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:15:52.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:52.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:52.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:53.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:53.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:53.942 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:15:53.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:15:53.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:53.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:54.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:55.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:55.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:55.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:15:55.330 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:15:55.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:55.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:55.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:56.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:56.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:56.624 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:15:56.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:15:56.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:56.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:56.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:57.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:57.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:57.926 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:15:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:15:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:58.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:15:59.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:15:59.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:15:59.343 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:15:59.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:15:59.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:15:59.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:15:59.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:00.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:00.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:00.795 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:16:00.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:16:00.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:00.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:16:01.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:16:02.570 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:16:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:16:03.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:04.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:04.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:04.431 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:16:04.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:16:04.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:04.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:16:04.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:05.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:05.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:05.778 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:16:05.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:16:05.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:05.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:16:06.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:07.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:07.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:07.241 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:16:07.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:16:07.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:07.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:16:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:08.965 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:16:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:16:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:16:09.254 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/638626470,v1:127.0.0.1:6803/638626470] [v2:127.0.0.1:6804/638626470,v1:127.0.0.1:6805/638626470] exists,up 63b3b5b3-cdd0-46fc-9312-d189f0118f60 2024-04-01T05:16:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:16:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:16:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:16:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-04-01T05:16:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:16:09.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:16:09.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:16:09.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad 2024-04-01T05:16:09.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad' 2024-04-01T05:16:09.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:16:09.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZQwpm17JwEBAAR1nX3cWtUu8zf8IEzeUPvA== 2024-04-01T05:16:09.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZQwpm17JwEBAAR1nX3cWtUu8zf8IEzeUPvA=="}' 2024-04-01T05:16:09.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad -i td/osd-scrub-repair/1/new.json 2024-04-01T05:16:09.673 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:16:09.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:16:09.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAZQwpm17JwEBAAR1nX3cWtUu8zf8IEzeUPvA== --osd-uuid bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad 2024-04-01T05:16:09.717 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.716+0000 7f14ae611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:09.758 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.757+0000 7f14ae611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:09.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.758+0000 7f14ae611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:09.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.758+0000 7f14ae611640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:16:09.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.758+0000 7f14ae611640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:16:09.759 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.758+0000 7f14ae611640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:16:09.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:09.832+0000 7f14ae611640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:16:15.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:16:15.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:16:15.575 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:16:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:16:15.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:16:15.941 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:16:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:16:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:16:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:16:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:16:15.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:16:15.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:15.970+0000 7f9c00415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:15.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:15.972+0000 7f9c00415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:15.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:15.973+0000 7f9c00415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:16:16.225 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:16:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:16.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:16.492+0000 7f9c00415640 -1 Falling back to public interface 2024-04-01T05:16:16.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:17.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:17.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:17.512 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:16:17.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:16:17.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:17.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:17.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:18.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:18.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:18.807 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:16:18.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:16:18.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:18.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:18.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:18.841+0000 7f9c00415640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:16:19.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:20.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:20.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:20.144 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:16:20.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:16:20.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:20.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:20.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:21.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:21.465 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:16:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:16:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:21.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:21.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:22.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:22.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:22.864 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:16:22.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:16:22.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:22.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:23.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:24.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:24.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:24.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:16:24.158 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:16:24.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:24.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:24.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:25.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:25.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:25.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:16:25.455 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:16:25.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:25.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:25.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:26.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:26.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:26.837 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:16:26.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:16:26.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:26.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:27.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:28.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:28.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:28.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:16:28.129 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:16:28.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:28.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:28.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:29.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:29.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:29.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:16:29.467 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:16:29.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:29.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:29.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:30.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:16:30.807 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:16:30.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:30.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:31.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:32.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:16:32.304 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:16:32.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:32.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:32.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:16:33.863 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:16:33.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:33.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:34.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:35.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:35.157 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:16:35.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:35.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:16:35.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:35.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:35.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:36.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:36.909 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:16:36.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:36.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:16:36.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:36.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:37.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:38.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:38.614 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:16:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:16:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:38.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:38.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:39.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:39.989 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:16:39.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:39.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:16:39.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:39.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:40.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:41.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:16:41.517 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:16:41.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:41.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:16:41.865 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2407890237,v1:127.0.0.1:6811/2407890237] [v2:127.0.0.1:6812/2407890237,v1:127.0.0.1:6813/2407890237] exists,up bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad 2024-04-01T05:16:41.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:16:41.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:16:41.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:16:41.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-04-01T05:16:41.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:16:41.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:16:41.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:16:41.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:16:41.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49ccef21-bdff-4a2d-a364-2c72e9abbbc3 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 49ccef21-bdff-4a2d-a364-2c72e9abbbc3' 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 49ccef21-bdff-4a2d-a364-2c72e9abbbc3 2024-04-01T05:16:41.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:16:41.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5Qwpm1VfWNBAAuMJ+FCqPqxUi9E2hPzobqw== 2024-04-01T05:16:41.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5Qwpm1VfWNBAAuMJ+FCqPqxUi9E2hPzobqw=="}' 2024-04-01T05:16:41.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49ccef21-bdff-4a2d-a364-2c72e9abbbc3 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:16:42.229 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:16:42.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:16:42.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA5Qwpm1VfWNBAAuMJ+FCqPqxUi9E2hPzobqw== --osd-uuid 49ccef21-bdff-4a2d-a364-2c72e9abbbc3 2024-04-01T05:16:42.273 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.273+0000 7f26af28b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.314+0000 7f26af28b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:42.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.315+0000 7f26af28b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:42.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.315+0000 7f26af28b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:16:42.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.316+0000 7f26af28b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:16:42.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.316+0000 7f26af28b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:16:42.380 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:42.380+0000 7f26af28b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:16:48.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:16:48.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:16:48.652 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:16:48.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:16:48.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:16:49.018 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:16:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:16:49.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:16:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:16:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:16:49.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:16:49.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:49.048+0000 7f9203e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:49.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:49.050+0000 7f9203e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:49.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:49.051+0000 7f9203e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:16:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:16:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:16:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:16:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:16:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:16:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:49.307 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:16:49.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:16:49.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:49.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:49.576 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:49.575+0000 7f9203e13640 -1 Falling back to public interface 2024-04-01T05:16:49.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:50.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:50.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:50.589 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:16:50.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:16:50.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:50.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:50.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:51.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:51.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:51.880 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:16:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:16:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:51.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:16:51.942+0000 7f9203e13640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:16:52.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:53.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:53.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:53.219 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:16:53.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:16:53.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:53.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:53.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:54.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:54.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:54.640 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:16:54.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:16:54.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:54.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:54.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:55.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:56.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:56.033 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:16:56.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:16:56.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:56.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:56.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:57.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:57.274 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:16:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:16:57.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:57.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:57.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:16:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:16:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:16:58.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:16:58.746 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:16:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:16:58.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:16:59.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:00.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:00.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:00.055 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:17:00.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:17:00.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:00.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:00.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:01.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:01.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:01.347 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:17:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:17:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:01.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:01.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:02.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:02.731 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:17:02.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:17:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:02.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:03.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:04.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:04.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:04.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:17:04.025 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:17:04.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:04.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:04.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:05.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:05.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:05.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:17:05.320 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:17:05.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:05.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:06.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:07.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:07.053 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:17:07.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:17:07.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:07.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:07.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:08.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:08.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:08.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:17:08.656 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:17:08.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:08.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:09.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:10.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:10.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:10.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:17:10.508 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:17:10.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:10.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:11.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:12.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:12.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:12.066 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:17:12.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:17:12.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:12.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:12.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:13.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:13.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:13.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:17:13.697 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:17:13.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:13.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:14.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:15.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:15.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:15.184 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:17:15.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:17:15.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:15.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:17:15.477 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/441448436,v1:127.0.0.1:6819/441448436] [v2:127.0.0.1:6820/441448436,v1:127.0.0.1:6821/441448436] exists,up 49ccef21-bdff-4a2d-a364-2c72e9abbbc3 2024-04-01T05:17:15.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:17:15.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:17:15.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:17:15.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-04-01T05:17:15.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:17:15.752 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:17:15.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:17:15.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:17:16.145 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:17:16.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:17:17.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:17:19.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-04-01T05:17:19.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:17:19.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:17:19.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:17:19.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:17:19.055 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:17:19.055 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:17:19.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:17:19.056 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:17:19.056 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:17:19.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:17:19.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:17:19.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:17:19.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:17:19.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:17:19.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:17:19.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:17:19.521 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:19.521 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:17:19.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:17:19.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:19.522 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:17:19.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-04-01T05:17:19.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-04-01T05:17:19.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-04-01T05:17:19.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:19.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:17:19.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-04-01T05:17:19.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-04-01T05:17:19.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-64424509451' 2024-04-01T05:17:19.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:19.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:17:19.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953477 2024-04-01T05:17:19.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953477 2024-04-01T05:17:19.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-64424509451 2-137438953477' 2024-04-01T05:17:19.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:19.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-04-01T05:17:19.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:19.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:17:19.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-04-01T05:17:19.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:19.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-04-01T05:17:19.849 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836501 2024-04-01T05:17:19.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-04-01T05:17:19.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:20.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-04-01T05:17:20.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:17:21.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:17:21.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:21.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-04-01T05:17:21.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:21.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509451 2024-04-01T05:17:21.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:21.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:17:21.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509451 2024-04-01T05:17:21.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:21.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-04-01T05:17:21.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509451' 2024-04-01T05:17:21.436 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 64424509451 2024-04-01T05:17:21.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:17:21.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-01T05:17:21.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:21.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953477 2024-04-01T05:17:21.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:21.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:17:21.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953477 2024-04-01T05:17:21.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:21.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953477 2024-04-01T05:17:21.721 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 137438953477 2024-04-01T05:17:21.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953477' 2024-04-01T05:17:21.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:22.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953477 -lt 137438953477 2024-04-01T05:17:22.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:17:22.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:22.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:17:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:17:22.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:17:22.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:17:22.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:17:22.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:17:22.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:17:22.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:17:22.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:17:22.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:17:22.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:22.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:17:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:17:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:17:23.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-04-01T05:17:23.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:17:23.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:17:23.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-04-01T05:17:23.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:17:23.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-01T05:17:23.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:17:23.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:17:24.164 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:17:24.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:17:25.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-04-01T05:17:25.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-04-01T05:17:25.801 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 2 allow_ec_overwrites to true 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:17:25.813 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:17:25.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:17:25.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:17:25.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:17:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:17:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:17:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:17:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:17:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:17:25.897 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:17:26.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:17:26.286 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:26.286 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:17:26.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:17:26.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:26.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:17:26.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-04-01T05:17:26.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-04-01T05:17:26.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-04-01T05:17:26.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:26.440 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:17:26.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-04-01T05:17:26.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-04-01T05:17:26.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-64424509454' 2024-04-01T05:17:26.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:26.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:17:26.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953480 2024-04-01T05:17:26.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953480 2024-04-01T05:17:26.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-64424509454 2-137438953480' 2024-04-01T05:17:26.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:26.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-04-01T05:17:26.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:26.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:17:26.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-04-01T05:17:26.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:26.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-04-01T05:17:26.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-01T05:17:26.658 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836504 2024-04-01T05:17:26.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:26.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-04-01T05:17:26.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:17:27.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:17:27.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:28.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-04-01T05:17:28.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:17:29.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:17:29.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:29.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-01T05:17:29.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:29.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509454 2024-04-01T05:17:29.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:29.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:17:29.577 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509454 2024-04-01T05:17:29.577 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:29.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-04-01T05:17:29.578 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 64424509454 2024-04-01T05:17:29.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509454' 2024-04-01T05:17:29.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:17:29.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-01T05:17:29.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:29.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953480 2024-04-01T05:17:29.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:29.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:17:29.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953480 2024-04-01T05:17:29.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:29.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953480 2024-04-01T05:17:29.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953480' 2024-04-01T05:17:29.902 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 137438953480 2024-04-01T05:17:29.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:30.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953480 -lt 137438953480 2024-04-01T05:17:30.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:17:30.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:30.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:17:30.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:17:30.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:17:30.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:17:30.841 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:30.841 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:31.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:17:31.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:17:31.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:17:31.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:17:31.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-04-01T05:17:31.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-04-01T05:17:31.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:17:31.364 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:17:31.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-04-01T05:17:31.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:17:31.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-04-01T05:17:31.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:17:31.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:17:31.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:17:31.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:17:31.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:17:31.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:17:31.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:17:33.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:17:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:17:34.589 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-04-01T05:17:34.589 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:17:35.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:17:35.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:17:35.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:17:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:17:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:17:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:17:35.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:17:35.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:35.139+0000 7f12fa20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:17:35.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:35.141+0000 7f12fa20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:17:35.143 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:35.143+0000 7f12fa20c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:17:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:17:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:17:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:17:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:17:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:17:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:17:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:17:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:35.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:35.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:35.657+0000 7f12fa20c640 -1 Falling back to public interface 2024-04-01T05:17:35.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:36.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:36.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:36.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:17:36.688 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:36.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:36.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:17:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:17:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:38.022 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:38.021+0000 7f12fa20c640 -1 osd.1 53 log_to_monitors true 2024-04-01T05:17:38.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:39.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:39.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:39.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:17:39.269 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:17:39.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:39.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:39.669 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 60 up_thru 60 down_at 54 last_clean_interval [15,53) [v2:127.0.0.1:6810/3756969314,v1:127.0.0.1:6811/3756969314] [v2:127.0.0.1:6812/3756969314,v1:127.0.0.1:6813/3756969314] exists,up bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad 2024-04-01T05:17:39.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:17:39.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:17:39.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:17:39.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:17:39.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:17:39.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:17:39.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:17:39.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:17:40.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:17:40.034 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:40.034 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:17:40.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:17:40.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:40.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:17:40.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-04-01T05:17:40.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-04-01T05:17:40.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-04-01T05:17:40.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:40.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:17:40.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-04-01T05:17:40.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-04-01T05:17:40.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-257698037762' 2024-04-01T05:17:40.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:40.274 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:17:40.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953484 2024-04-01T05:17:40.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953484 2024-04-01T05:17:40.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-257698037762 2-137438953484' 2024-04-01T05:17:40.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:40.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-04-01T05:17:40.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:40.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:17:40.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-04-01T05:17:40.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:40.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-04-01T05:17:40.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-01T05:17:40.382 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836508 2024-04-01T05:17:40.383 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:40.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-01T05:17:40.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:40.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037762 2024-04-01T05:17:40.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:40.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:17:40.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037762 2024-04-01T05:17:40.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:40.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-04-01T05:17:40.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2024-04-01T05:17:40.672 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 257698037762 2024-04-01T05:17:40.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:17:40.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-04-01T05:17:40.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:40.962 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953484 2024-04-01T05:17:40.962 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:40.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:17:40.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953484 2024-04-01T05:17:40.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:40.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953484 2024-04-01T05:17:40.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953484' 2024-04-01T05:17:40.965 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 137438953484 2024-04-01T05:17:40.965 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953482 -lt 137438953484 2024-04-01T05:17:41.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:17:42.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:17:42.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953484 -lt 137438953484 2024-04-01T05:17:42.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:17:42.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:42.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:42.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:17:42.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:17:42.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:17:42.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:17:42.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:17:42.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:17:42.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:17:42.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:17:43.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:17:43.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:17:43.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:43.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:43.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:17:43.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:17:43.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:17:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-04-01T05:17:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:17:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:17:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:17:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:17:43.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-04-01T05:17:43.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-04-01T05:17:43.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:17:43.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:17:43.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:17:43.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:17:44.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-04-01T05:17:24.038916+0000 2024-04-01T05:17:44.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:17:44.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:17:24.038916+0000 2024-04-01T05:17:44.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:17:44.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:17:44.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:17:44.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:17:44.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:17:44.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:17:44.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:17:44.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:17:44.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:17:24.038916+0000 '>' 2024-04-01T05:17:24.038916+0000 2024-04-01T05:17:44.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:17:45.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:17:45.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:17:45.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:17:45.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:17:45.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:17:45.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:17:45.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:17:42.011841+0000 '>' 2024-04-01T05:17:24.038916+0000 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:17:45.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:17:45.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:17:45.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:17:45.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:17:45.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:17:45.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:17:45.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:17:46.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:17:46.110 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:46.110 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:17:46.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:17:46.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:46.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:17:46.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-04-01T05:17:46.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-04-01T05:17:46.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-04-01T05:17:46.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:46.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:17:46.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037764 2024-04-01T05:17:46.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037764 2024-04-01T05:17:46.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-257698037764' 2024-04-01T05:17:46.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:46.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:17:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953486 2024-04-01T05:17:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953486 2024-04-01T05:17:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-257698037764 2-137438953486' 2024-04-01T05:17:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:46.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-04-01T05:17:46.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:46.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:17:46.461 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-04-01T05:17:46.461 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:46.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-04-01T05:17:46.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-01T05:17:46.462 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836510 2024-04-01T05:17:46.462 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:46.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-01T05:17:46.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:46.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-257698037764 2024-04-01T05:17:46.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:46.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:17:46.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-257698037764 2024-04-01T05:17:46.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:46.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037764 2024-04-01T05:17:46.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 257698037764' 2024-04-01T05:17:46.746 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 257698037764 2024-04-01T05:17:46.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:17:47.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037764 -lt 257698037764 2024-04-01T05:17:47.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:47.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953486 2024-04-01T05:17:47.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:47.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:17:47.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953486 2024-04-01T05:17:47.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:47.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953486 2024-04-01T05:17:47.031 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 137438953486 2024-04-01T05:17:47.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953486' 2024-04-01T05:17:47.031 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:47.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953485 -lt 137438953486 2024-04-01T05:17:47.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:17:48.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:17:48.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:48.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953486 -lt 137438953486 2024-04-01T05:17:48.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:17:48.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:48.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:48.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:17:48.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:17:49.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:17:49.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:17:49.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:49.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:49.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:17:49.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:17:49.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:17:49.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-04-01T05:17:49.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-04-01T05:17:49.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:17:49.618 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-04-01T05:17:49.618 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:17:49.618 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:17:49.618 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:17:49.618 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:17:49.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-04-01T05:17:49.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:17:49.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:17:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:17:50.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:17:50.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:17:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:17:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:17:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:17:50.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:17:50.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:17:50.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:17:50.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:17:50.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:17:50.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:17:50.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:17:50.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:17:52.076 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:17:52.077 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-04-01T05:17:52.077 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:17:52.077 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:17:52.077 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:17:52.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:17:52.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:17:52.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:17:52.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-04-01T05:17:52.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:52.390+0000 7f0050611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:52.404+0000 7f0050611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:17:52.405 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:52.405+0000 7f0050611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:17:52.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:52.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:52.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:52.923+0000 7f0050611640 -1 Falling back to public interface 2024-04-01T05:17:52.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:17:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:54.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:17:55.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:17:55.069+0000 7f0050611640 -1 osd.1 66 log_to_monitors true 2024-04-01T05:17:55.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:17:55.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:17:55.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:17:55.232 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:17:55.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:17:55.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:17:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 72 up_thru 60 down_at 67 last_clean_interval [60,66) [v2:127.0.0.1:6810/1141152087,v1:127.0.0.1:6811/1141152087] [v2:127.0.0.1:6812/1141152087,v1:127.0.0.1:6813/1141152087] exists,up bd09bbf4-f6c1-49f6-8a5b-0468ad3069ad 2024-04-01T05:17:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:17:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:17:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:17:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:17:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:17:55.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:17:55.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:17:55.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:17:55.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:17:55.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:17:55.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:17:55.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:17:56.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:17:56.218 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:17:56.218 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:17:56.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:17:56.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:56.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:17:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-04-01T05:17:56.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-04-01T05:17:56.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-04-01T05:17:56.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:56.412 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:17:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-04-01T05:17:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-04-01T05:17:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-309237645314' 2024-04-01T05:17:56.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:17:56.663 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953489 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953489 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-309237645314 2-137438953489' 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:56.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:17:56.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-04-01T05:17:56.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:56.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-04-01T05:17:56.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-04-01T05:17:56.850 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836513 2024-04-01T05:17:56.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-04-01T05:17:57.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:17:58.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:17:58.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:17:58.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-04-01T05:17:58.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:58.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645314 2024-04-01T05:17:58.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:58.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:17:58.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645314 2024-04-01T05:17:58.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:58.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-04-01T05:17:58.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2024-04-01T05:17:58.508 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 309237645314 2024-04-01T05:17:58.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:17:58.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-04-01T05:17:58.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:17:58.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953489 2024-04-01T05:17:58.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:17:58.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:17:58.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953489 2024-04-01T05:17:58.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:17:58.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953489 2024-04-01T05:17:58.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953489' 2024-04-01T05:17:58.796 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 137438953489 2024-04-01T05:17:58.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:17:59.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953489 -lt 137438953489 2024-04-01T05:17:59.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:17:59.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:59.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:17:59.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:17:59.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:17:59.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:17:59.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:17:59.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:17:59.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:17:59.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:17:59.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:17:59.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:17:59.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:17:59.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:17:59.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:18:00.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:18:00.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:18:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:18:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:18:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:18:00.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:18:00.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:18:00.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:18:00.547 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:18:00.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:18:00.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:18:00.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:18:00.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:18:00.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:18:01.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:18:01.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:18:01.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:18:01.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:18:01.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:18:01.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:18:01.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:18:01.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:18:01.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:18:01.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:18:01.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:18:01.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:18:01.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:18:01.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:18:01.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:18:01.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:18:01.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:18:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:18:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:18:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:18:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:18:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:18:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:18:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:18:02.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:18:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:18:02.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:18:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:18:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:18:02.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:18:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:18:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:18:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:18:02.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:18:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:18:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:18:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:18:02.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:18:02.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:02.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:02.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:18:02.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:18:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:18:02.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:18:02.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:18:02.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:18:02.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:18:02.399 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:18:02.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:18:02.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:18:02.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:18:02.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:18:02.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:18:02.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:18:02.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:18:02.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:18:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:18:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:18:02.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:18:02.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:02.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:02.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:18:02.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:18:02.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:18:02.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:18:02.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:18:02.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:02.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:02.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:18:02.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:18:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:18:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:18:02.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:18:02.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:18:02.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:18:02.790 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:18:02.790 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:02.790 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:02.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:18:02.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:18:02.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:18:02.983 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:18:02.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:18:02.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:18:02.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:18:02.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:18:02.984 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:18:02.984 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:18:02.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:18:03.067 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:18:03.068 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:18:03.068 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:03.068 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:03.068 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:18:03.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:18:03.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:18:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-04-01T05:18:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:18:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:18:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:18:03.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:18:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:18:03.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:18:03.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:18:03.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:18:03.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:18:03.943 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:18:03.943 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:03.943 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:03.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:18:03.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:18:03.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:18:03.970 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:18:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:18:03.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:18:03.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:18:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70715bbc-e452-4a0e-a473-f5df52c0254b 2024-04-01T05:18:03.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 70715bbc-e452-4a0e-a473-f5df52c0254b' 2024-04-01T05:18:03.977 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 70715bbc-e452-4a0e-a473-f5df52c0254b 2024-04-01T05:18:03.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:18:03.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLQwpm4ksfOxAAg2kKp/dL9fLa/5anX3O4SA== 2024-04-01T05:18:03.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLQwpm4ksfOxAAg2kKp/dL9fLa/5anX3O4SA=="}' 2024-04-01T05:18:03.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70715bbc-e452-4a0e-a473-f5df52c0254b -i td/osd-scrub-repair/0/new.json 2024-04-01T05:18:04.198 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:18:04.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:18:04.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLQwpm4ksfOxAAg2kKp/dL9fLa/5anX3O4SA== --osd-uuid 70715bbc-e452-4a0e-a473-f5df52c0254b 2024-04-01T05:18:04.241 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.241+0000 7f3d2d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:04.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.249+0000 7f3d2d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:04.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.250+0000 7f3d2d20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:04.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.251+0000 7f3d2d20d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:18:04.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.251+0000 7f3d2d20d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:18:04.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.251+0000 7f3d2d20d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:18:04.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:04.332+0000 7f3d2d20d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:18:09.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:18:09.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:18:09.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:18:09.928 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:18:09.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:18:10.335 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:18:10.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:18:10.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:18:10.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:18:10.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:18:10.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:18:10.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:10.361+0000 7fe67181e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:10.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:10.363+0000 7fe67181e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:10.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:10.365+0000 7fe67181e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:18:10.623 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:18:10.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:10.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:10.882 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:10.882+0000 7fe67181e640 -1 Falling back to public interface 2024-04-01T05:18:10.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:11.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:11.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:11.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:18:11.957 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:18:11.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:11.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:12.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:13.224 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:13.224+0000 7fe67181e640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:18:13.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:13.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:13.249 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:18:13.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:18:13.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:13.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:13.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:14.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:14.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:14.659 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:18:14.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:18:14.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:14.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:14.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:15.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:15.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:16.000 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:18:16.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:18:16.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:16.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:16.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:17.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:17.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:17.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:18:17.297 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:18:17.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:17.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:17.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:18.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:18.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:18.590 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:18:18.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:18:18.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:18.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:18.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:19.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:19.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:19.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:18:19.948 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:18:19.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:19.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:20.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:21.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:21.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:21.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:18:21.243 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:18:21.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:21.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:21.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:22.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:22.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:22.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:18:22.540 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:18:22.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:22.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:22.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:23.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:23.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:23.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:18:23.836 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:18:23.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:23.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:24.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:25.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:25.346 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:18:25.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:18:25.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:25.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:25.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:26.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:26.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:26.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:18:26.979 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:18:26.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:26.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:27.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:28.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:28.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:28.297 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:18:28.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:18:28.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:28.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:29.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:30.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:30.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:30.059 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:18:30.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:18:30.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:30.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:30.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:31.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:31.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:31.450 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:18:31.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:18:31.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:31.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:33.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:33.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:33.114 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:18:33.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:18:33.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:33.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:33.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:34.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:34.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:34.644 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:18:34.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:18:34.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:34.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:18:34.935 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1722566086,v1:127.0.0.1:6803/1722566086] [v2:127.0.0.1:6804/1722566086,v1:127.0.0.1:6805/1722566086] exists,up 70715bbc-e452-4a0e-a473-f5df52c0254b 2024-04-01T05:18:34.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:18:34.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:18:34.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:18:34.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:18:34.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:18:34.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 18bd570f-9be1-4dc3-9b20-bc20cffbb88e' 2024-04-01T05:18:34.942 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:18:34.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:18:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqQwpmymoaORAAy0AIXh53ze+MXRq4B/R0kQ== 2024-04-01T05:18:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqQwpmymoaORAAy0AIXh53ze+MXRq4B/R0kQ=="}' 2024-04-01T05:18:34.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 18bd570f-9be1-4dc3-9b20-bc20cffbb88e -i td/osd-scrub-repair/1/new.json 2024-04-01T05:18:35.394 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:18:35.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:18:35.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqQwpmymoaORAAy0AIXh53ze+MXRq4B/R0kQ== --osd-uuid 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:18:35.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.433+0000 7fa992536640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:35.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.478+0000 7fa992536640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:35.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.479+0000 7fa992536640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:35.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.480+0000 7fa992536640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:18:35.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.480+0000 7fa992536640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:18:35.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.480+0000 7fa992536640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:18:35.574 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:35.574+0000 7fa992536640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:18:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:18:41.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:18:41.251 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:18:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:18:41.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:18:41.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:18:41.628 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:18:41.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:18:41.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:18:41.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:18:41.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:18:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:41.654+0000 7f7314d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:41.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:41.655+0000 7f7314d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:41.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:41.657+0000 7f7314d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:41.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:18:41.911 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:18:41.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:41.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:42.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:42.186+0000 7f7314d38640 -1 Falling back to public interface 2024-04-01T05:18:42.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:43.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:43.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:43.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:18:43.194 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:18:43.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:43.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:43.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:44.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:44.486 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:18:44.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:44.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:18:44.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:44.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:44.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:18:44.569+0000 7f7314d38640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:18:44.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:45.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:45.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:45.781 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:18:45.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:18:45.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:45.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:46.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:47.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:47.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:47.074 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:18:47.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:18:47.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:47.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:47.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:48.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:48.580 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:18:48.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:18:48.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:48.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:48.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:49.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:49.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:49.876 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:18:49.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:18:49.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:49.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:50.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:51.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:51.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:51.171 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:18:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:18:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:51.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:52.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:52.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:52.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:18:52.588 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:18:52.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:52.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:52.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:53.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:53.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:53.885 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:18:53.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:18:53.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:53.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:55.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:55.179 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:18:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:18:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:55.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:55.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:56.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:56.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:56.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:18:56.523 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:18:56.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:56.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:18:58.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:18:58.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:18:58.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:18:58.266 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:18:58.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:18:58.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:18:59.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:00.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:00.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:00.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:19:00.365 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:19:00.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:00.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:19:01.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:02.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:02.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:02.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:19:02.049 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:19:02.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:02.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:19:02.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:03.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:03.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:03.861 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:19:03.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:19:03.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:03.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:19:04.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:05.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:05.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:05.672 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:19:05.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:19:05.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:05.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:19:06.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:07.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:07.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:07.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:19:07.359 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:19:07.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:07.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:19:07.652 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/128667637,v1:127.0.0.1:6811/128667637] [v2:127.0.0.1:6812/128667637,v1:127.0.0.1:6813/128667637] exists,up 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:19:07.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:19:07.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:19:07.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:19:07.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:19:07.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:19:07.665 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:19:07.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0be321f-1cd9-41dc-a51d-5e81df352119 2024-04-01T05:19:07.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d0be321f-1cd9-41dc-a51d-5e81df352119' 2024-04-01T05:19:07.665 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 d0be321f-1cd9-41dc-a51d-5e81df352119 2024-04-01T05:19:07.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:19:07.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLQwpmNF0fKBAAkMcCYtzJDWRXAL8BQMc+OQ== 2024-04-01T05:19:07.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLQwpmNF0fKBAAkMcCYtzJDWRXAL8BQMc+OQ=="}' 2024-04-01T05:19:07.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0be321f-1cd9-41dc-a51d-5e81df352119 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:19:08.007 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:19:08.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:19:08.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLQwpmNF0fKBAAkMcCYtzJDWRXAL8BQMc+OQ== --osd-uuid d0be321f-1cd9-41dc-a51d-5e81df352119 2024-04-01T05:19:08.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.047+0000 7f571456b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:08.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.091+0000 7f571456b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.092+0000 7f571456b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:08.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.093+0000 7f571456b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:19:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.093+0000 7f571456b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:19:08.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.093+0000 7f571456b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:19:08.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:08.158+0000 7f571456b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:19:13.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:19:13.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:19:13.867 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:19:13.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:19:13.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:19:14.233 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:19:14.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:19:14.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:19:14.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:19:14.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:19:14.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:19:14.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:14.259+0000 7f6729e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:14.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:14.262+0000 7f6729e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:14.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:14.263+0000 7f6729e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:14.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:19:14.520 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:19:14.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:14.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:14.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:14.790+0000 7f6729e17640 -1 Falling back to public interface 2024-04-01T05:19:14.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:19:15.803 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:19:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:15.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:16.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:17.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:17.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:19:17.096 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:19:17.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:17.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:17.174 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:17.174+0000 7f6729e17640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:19:17.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:18.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:18.423 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:19:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:19:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:18.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:18.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:19.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:19.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:19.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:19:19.718 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:19:19.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:19.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:20.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:21.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:21.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:21.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:19:21.122 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:19:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:21.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:21.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:22.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:22.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:22.461 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:19:22.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:19:22.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:22.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:22.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:23.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:23.754 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:19:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:19:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:23.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:24.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:25.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:25.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:25.081 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:19:25.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:19:25.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:25.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:26.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:26.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:26.389 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:19:26.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:19:26.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:26.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:26.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:27.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:27.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:27.683 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:19:27.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:19:27.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:27.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:27.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:28.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:28.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:28.977 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:19:28.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:19:28.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:28.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:29.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:30.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:30.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:30.511 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:19:30.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:19:30.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:30.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:31.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:32.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:32.454 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:19:32.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:19:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:32.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:33.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:34.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:34.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:34.036 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:19:34.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:19:34.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:34.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:34.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:35.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:35.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:35.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:19:35.584 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:19:35.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:35.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:36.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:37.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:37.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:37.344 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:19:37.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:19:37.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:37.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:37.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:38.812 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:19:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:19:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:38.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:39.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:40.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:40.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:40.610 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:19:40.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:19:40.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:40.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:19:40.904 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/771235162,v1:127.0.0.1:6819/771235162] [v2:127.0.0.1:6820/771235162,v1:127.0.0.1:6821/771235162] exists,up d0be321f-1cd9-41dc-a51d-5e81df352119 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:19:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:19:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:19:40.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:19:40.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:19:40.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:19:40.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:19:40.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 23c8b24d-b8b4-4ed0-b072-b5f9f063108d' 2024-04-01T05:19:40.911 INFO:tasks.workunit.client.0.smithi067.stdout:add osd3 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:19:40.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:19:40.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsQwpm1/4rNxAALgDglxIDrawIbwUTCF/rMg== 2024-04-01T05:19:40.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsQwpm1/4rNxAALgDglxIDrawIbwUTCF/rMg=="}' 2024-04-01T05:19:40.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 23c8b24d-b8b4-4ed0-b072-b5f9f063108d -i td/osd-scrub-repair/3/new.json 2024-04-01T05:19:41.287 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:19:41.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-04-01T05:19:41.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsQwpm1/4rNxAALgDglxIDrawIbwUTCF/rMg== --osd-uuid 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:19:41.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.327+0000 7f679020c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:41.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.372+0000 7f679020c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:41.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.373+0000 7f679020c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:41.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.374+0000 7f679020c640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:19:41.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.374+0000 7f679020c640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:19:41.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.374+0000 7f679020c640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:19:41.475 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:41.476+0000 7f679020c640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-04-01T05:19:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-04-01T05:19:47.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:19:47.178 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd3 key to auth repository 2024-04-01T05:19:47.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-01T05:19:47.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:19:47.544 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.3 2024-04-01T05:19:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-01T05:19:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:19:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:19:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:19:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:19:47.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:47.569+0000 7fa2d1c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:47.572 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:47.571+0000 7fa2d1c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:47.573 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:47.572+0000 7fa2d1c17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:19:47.828 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:19:47.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:47.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:48.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:48.087+0000 7fa2d1c17640 -1 Falling back to public interface 2024-04-01T05:19:48.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:49.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:49.107 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:19:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:19:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:49.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:50.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:50.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:50.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:19:50.400 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:19:50.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:50.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:50.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:19:50.421+0000 7fa2d1c17640 -1 osd.3 0 log_to_monitors true 2024-04-01T05:19:50.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:51.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:51.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:51.722 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:19:51.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:19:51.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:51.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:52.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:53.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:53.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:53.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:19:53.017 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:19:53.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:53.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:53.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:54.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:54.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:54.371 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:19:54.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:19:54.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:54.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:54.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:55.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:55.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:55.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:19:55.666 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:19:55.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:55.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:55.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:56.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:56.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:56.983 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:19:56.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:19:56.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:56.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:57.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:58.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:58.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:58.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:19:58.352 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:19:58.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:58.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:58.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:19:59.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:19:59.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:19:59.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:19:59.647 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:19:59.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:19:59.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:19:59.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:00.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:00.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:00.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:20:00.964 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:20:00.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:00.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:01.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:02.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:02.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:02.257 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:20:02.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:20:02.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:02.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:03.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:03.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:03.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:20:03.841 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:20:03.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:03.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:04.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:05.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:20:05.443 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:20:05.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:05.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:05.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:06.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:06.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:06.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:20:06.934 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:20:06.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:06.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:07.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:08.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:08.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:08.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:20:08.749 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:20:08.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:08.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:09.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:10.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:10.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:10.470 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:20:10.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:20:10.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:10.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:10.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:11.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:11.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:11.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:20:11.963 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:20:11.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:11.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:12.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:13.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:13.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:13.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:20:13.429 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:20:13.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:13.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:13.720 INFO:tasks.workunit.client.0.smithi067.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1718926841,v1:127.0.0.1:6827/1718926841] [v2:127.0.0.1:6828/1718926841,v1:127.0.0.1:6829/1718926841] exists,up 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:20:13.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:20:13.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:20:13.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:20:13.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-04-01T05:20:13.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:20:13.996 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:20:14.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:20:14.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:20:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:20:14.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:20:15.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:20:17.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-04-01T05:20:17.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:20:17.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:20:17.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:20:17.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:20:17.655 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:20:17.656 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:20:17.656 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:20:17.656 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:20:17.656 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:20:17.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:20:17.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:20:17.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:20:17.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:20:17.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:20:17.750 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:18.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:20:18.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-04-01T05:20:18.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-04-01T05:20:18.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-04-01T05:20:18.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:18.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:20:18.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-04-01T05:20:18.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-04-01T05:20:18.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672983' 2024-04-01T05:20:18.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:18.244 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:20:18.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-04-01T05:20:18.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-04-01T05:20:18.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672983 2-64424509453' 2024-04-01T05:20:18.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:18.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:20:18.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345930 2024-04-01T05:20:18.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345930 2024-04-01T05:20:18.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672983 2-64424509453 3-85899345930' 2024-04-01T05:20:18.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:18.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-04-01T05:20:18.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:18.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:20:18.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-04-01T05:20:18.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:18.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-04-01T05:20:18.457 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836502 2024-04-01T05:20:18.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-01T05:20:18.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:18.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-04-01T05:20:18.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:20:19.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:20:19.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:20.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-04-01T05:20:20.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:20:21.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:20:21.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:21.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836502 2024-04-01T05:20:21.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:21.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-04-01T05:20:21.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:21.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:20:21.343 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-04-01T05:20:21.343 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:21.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-04-01T05:20:21.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-01T05:20:21.345 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672983 2024-04-01T05:20:21.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:20:21.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-01T05:20:21.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:21.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-04-01T05:20:21.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:20:21.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-04-01T05:20:21.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-04-01T05:20:21.629 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509453 2024-04-01T05:20:21.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-04-01T05:20:21.630 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:20:21.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-04-01T05:20:21.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:21.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345930 2024-04-01T05:20:21.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:21.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:20:21.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345930 2024-04-01T05:20:21.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:21.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345930 2024-04-01T05:20:21.914 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345930 2024-04-01T05:20:21.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-04-01T05:20:21.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:20:22.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345930 -lt 85899345930 2024-04-01T05:20:22.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:20:22.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:22.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:22.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:20:22.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:20:22.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:20:22.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:20:22.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:20:22.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:20:22.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:20:22.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:20:22.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:20:22.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:20:22.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:22.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:20:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:20:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:20:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-04-01T05:20:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:20:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:20:23.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-04-01T05:20:23.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:20:23.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-04-01T05:20:23.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:20:23.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:20:24.298 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:20:24.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:20:25.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-04-01T05:20:25.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:20:25.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:20:25.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:20:25.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:20:25.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:20:25.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:20:25.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:20:25.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:20:25.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:20:25.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:20:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:20:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:20:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:20:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:20:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:20:25.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:25.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:20:25.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-04-01T05:20:25.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-04-01T05:20:25.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-04-01T05:20:25.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:25.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:20:26.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-04-01T05:20:26.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-04-01T05:20:26.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672985' 2024-04-01T05:20:26.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:26.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:20:26.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-04-01T05:20:26.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-04-01T05:20:26.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509456' 2024-04-01T05:20:26.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:26.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:20:26.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-04-01T05:20:26.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-04-01T05:20:26.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509456 3-85899345932' 2024-04-01T05:20:26.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:26.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-04-01T05:20:26.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:26.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:20:26.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-04-01T05:20:26.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:26.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-04-01T05:20:26.215 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836505 2024-04-01T05:20:26.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-01T05:20:26.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:26.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-01T05:20:26.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:26.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-04-01T05:20:26.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:26.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:20:26.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-04-01T05:20:26.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:26.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-04-01T05:20:26.504 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672985 2024-04-01T05:20:26.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-01T05:20:26.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:20:26.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-01T05:20:26.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:26.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-04-01T05:20:26.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:26.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:20:26.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-04-01T05:20:26.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-04-01T05:20:26.816 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509456 2024-04-01T05:20:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-01T05:20:26.817 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:20:27.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509456 2024-04-01T05:20:27.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:20:28.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:20:28.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:20:28.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-04-01T05:20:28.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:28.399 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-04-01T05:20:28.399 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:28.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:20:28.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-04-01T05:20:28.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:28.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-04-01T05:20:28.401 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345932 2024-04-01T05:20:28.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-04-01T05:20:28.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:20:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345933 -lt 85899345932 2024-04-01T05:20:28.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:20:28.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:28.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:29.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:20:29.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:20:29.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:20:29.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:20:29.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:20:29.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:20:29.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:20:29.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:20:29.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:20:29.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:20:29.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:29.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:20:29.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-04-01T05:20:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:20:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:20:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:20:30.185 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:20:30.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:20:30.713 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:20:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:20:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:20:30.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:20:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-04-01T05:20:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:20:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:20:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:20:30.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:20:31.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-04-01T05:20:31.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-04-01T05:20:31.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T05:20:31.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-01T05:20:31.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-04-01T05:20:31.267 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:20:31.267 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-04-01T05:20:31.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-04-01T05:20:31.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:20:31.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:20:31.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:20:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:20:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:20:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:20:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:20:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:20:31.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:20:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:20:31.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:20:33.423 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:20:33.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:20:33.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:20:33.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:20:33.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:20:33.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:20:33.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:20:33.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:20:33.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:20:33.969 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:33.968+0000 7f5e388f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:20:33.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:33.971+0000 7f5e388f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:20:33.973 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:33.972+0000 7f5e388f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:20:34.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:34.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:34.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:34.487+0000 7f5e388f9640 -1 Falling back to public interface 2024-04-01T05:20:34.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:35.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:35.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:35.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:20:35.530 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:35.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:35.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:35.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:36.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:36.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:36.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:20:36.822 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:36.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:36.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:36.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:36.852+0000 7f5e388f9640 -1 osd.3 35 log_to_monitors true 2024-04-01T05:20:37.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:38.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:38.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:38.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:20:38.118 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:20:38.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:38.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/1120151605,v1:127.0.0.1:6827/1120151605] [v2:127.0.0.1:6828/1120151605,v1:127.0.0.1:6829/1120151605] exists,up 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:20:38.412 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:20:38.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:20:38.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:20:38.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:20:38.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:20:38.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:20:38.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:20:38.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:20:38.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:20:38.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:20:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:20:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:38.810 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:20:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:20:38.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:38.811 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:20:38.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-04-01T05:20:38.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-04-01T05:20:38.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-04-01T05:20:38.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:38.916 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:20:39.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-04-01T05:20:39.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-04-01T05:20:39.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672989' 2024-04-01T05:20:39.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:39.021 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:20:39.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-04-01T05:20:39.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-04-01T05:20:39.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672989 2-64424509459' 2024-04-01T05:20:39.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:39.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672989 2-64424509459 3-167503724546' 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:39.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:20:39.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-04-01T05:20:39.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:39.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-04-01T05:20:39.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-01T05:20:39.234 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836508 2024-04-01T05:20:39.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:39.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-04-01T05:20:39.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:20:40.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:20:40.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:40.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836508 2024-04-01T05:20:40.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:40.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-04-01T05:20:40.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:40.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:20:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-04-01T05:20:40.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:40.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-04-01T05:20:40.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-01T05:20:40.858 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672989 2024-04-01T05:20:40.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:20:41.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-01T05:20:41.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:41.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-04-01T05:20:41.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:41.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:20:41.162 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-04-01T05:20:41.162 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:41.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-04-01T05:20:41.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-04-01T05:20:41.163 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509459 2024-04-01T05:20:41.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:20:41.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-04-01T05:20:41.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:41.448 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-167503724546 2024-04-01T05:20:41.448 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:41.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:20:41.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-167503724546 2024-04-01T05:20:41.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:41.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-04-01T05:20:41.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 167503724546' 2024-04-01T05:20:41.451 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 167503724546 2024-04-01T05:20:41.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:20:41.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-04-01T05:20:41.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:20:41.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:41.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:20:42.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:20:42.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:20:42.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:20:42.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:20:42.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:42.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:42.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:20:42.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:20:42.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:20:42.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:20:42.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:20:42.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:20:42.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:20:42.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:20:43.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:20:43.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:20:43.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:43.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:20:43.556 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:20:43.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:20:43.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:20:43.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:20:43.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:20:43.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:20:43.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:24.207553+0000 '>' 2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:43.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:20:44.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:20:44.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:20:44.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:20:44.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:20:44.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:20:44.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:20:44.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:20:45.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:24.207553+0000 '>' 2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:45.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:20:46.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:20:46.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:20:46.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:20:46.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:20:46.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:20:46.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:20:46.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:20:46.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:24.207553+0000 '>' 2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:46.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:20:47.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:20:47.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:20:47.438 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:20:47.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:20:47.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:20:47.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:20:47.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:20:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:24.207553+0000 '>' 2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:20:48.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:20:48.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:20:48.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:20:48.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:20:48.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:20:48.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:20:48.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:20:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:43.762877+0000 '>' 2024-04-01T05:20:24.207553+0000 2024-04-01T05:20:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:20:49.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:20:49.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:20:49.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:20:49.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:20:49.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:20:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:20:50.938 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:20:50.939 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:20:50.939 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:20:50.939 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:20:51.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:20:51.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:20:51.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:20:51.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:20:51.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:20:51.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:20:51.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:20:51.212 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:20:51.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:20:51.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:20:51.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:20:51.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:20:51.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:20:51.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:20:51.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:51.238+0000 7f6f24224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:20:51.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:51.241+0000 7f6f24224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:20:51.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:51.242+0000 7f6f24224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:20:51.505 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:20:51.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:51.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:51.755 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:51.756+0000 7f6f24224640 -1 Falling back to public interface 2024-04-01T05:20:51.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:52.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:52.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:52.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:20:52.790 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:52.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:52.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:53.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:20:54.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:20:54.043+0000 7f6f24224640 -1 osd.3 41 log_to_monitors true 2024-04-01T05:20:54.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:20:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:20:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:20:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:20:54.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 46 up_thru 39 down_at 42 last_clean_interval [39,41) [v2:127.0.0.1:6826/1746091682,v1:127.0.0.1:6827/1746091682] [v2:127.0.0.1:6828/1746091682,v1:127.0.0.1:6829/1746091682] exists,up 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:20:54.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:20:54.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:20:54.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:20:54.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:20:54.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:20:54.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:20:54.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:20:54.587 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:20:54.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:20:54.947 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:20:54.947 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:20:54.947 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:20:54.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:20:54.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:54.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:20:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-04-01T05:20:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-04-01T05:20:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-04-01T05:20:55.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:55.190 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:20:55.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672993 2024-04-01T05:20:55.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672993 2024-04-01T05:20:55.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672993' 2024-04-01T05:20:55.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:55.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:20:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-04-01T05:20:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-04-01T05:20:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509463' 2024-04-01T05:20:55.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:20:55.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:20:55.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-04-01T05:20:55.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-04-01T05:20:55.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-64424509463 3-197568495618' 2024-04-01T05:20:55.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:55.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-04-01T05:20:55.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:55.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:20:55.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-04-01T05:20:55.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:55.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-04-01T05:20:55.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-04-01T05:20:55.677 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836513 2024-04-01T05:20:55.677 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:56.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836513 2024-04-01T05:20:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:20:57.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:20:57.023 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:20:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-04-01T05:20:57.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:57.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672993 2024-04-01T05:20:57.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:57.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:20:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672993 2024-04-01T05:20:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:57.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672993 2024-04-01T05:20:57.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-04-01T05:20:57.335 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672993 2024-04-01T05:20:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:20:57.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672993 -lt 42949672993 2024-04-01T05:20:57.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:57.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-04-01T05:20:57.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:57.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:20:57.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-04-01T05:20:57.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:57.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-04-01T05:20:57.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-04-01T05:20:57.617 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509463 2024-04-01T05:20:57.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:20:57.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509463 2024-04-01T05:20:57.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:20:57.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-04-01T05:20:57.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:20:57.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:20:57.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-04-01T05:20:57.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:20:57.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-04-01T05:20:57.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-04-01T05:20:57.902 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 197568495618 2024-04-01T05:20:57.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:20:58.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-04-01T05:20:58.184 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:20:58.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:58.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:58.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:20:58.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:20:58.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:20:58.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:20:58.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:20:58.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:20:58.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:20:58.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:20:58.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:20:58.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:20:58.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:58.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:59.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:20:59.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:20:59.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:20:59.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:20:59.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:20:59.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:20:59.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:20:59.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:20:59.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:20:59.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:20:59.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:20:59.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:20:59.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:21:00.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:21:00.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:21:00.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:21:00.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:21:00.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:21:00.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:21:00.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:00.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:00.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:00.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:00.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:00.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:00.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:00.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:21:00.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:00.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:00.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:01.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:21:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:21:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:21:01.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:21:01.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:21:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-04-01T05:21:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:21:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:21:01.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:21:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:21:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:21:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:21:01.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:21:01.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:21:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:21:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:21:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:21:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:21:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:21:02.912 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:21:03.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:21:03.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:21:03.438 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:21:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:21:03.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:21:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:21:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:21:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:21:03.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:21:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:03.466+0000 7fe4870f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:03.468+0000 7fe4870f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:03.469+0000 7fe4870f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:03.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:21:03.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:21:03.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:21:03.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:21:03.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:21:03.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:03.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:21:03.732 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:21:03.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:03.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:03.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:03.984+0000 7fe4870f9640 -1 Falling back to public interface 2024-04-01T05:21:04.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:05.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:05.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:05.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:21:05.015 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:21:05.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:05.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:05.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:21:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:21:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:06.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:06.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:06.443+0000 7fe4870f9640 -1 osd.1 48 log_to_monitors true 2024-04-01T05:21:06.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:07.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:07.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:07.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:21:07.657 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:21:07.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:07.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/2953677090,v1:127.0.0.1:6811/2953677090] [v2:127.0.0.1:6812/2953677090,v1:127.0.0.1:6813/2953677090] exists,up 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:21:08.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:21:08.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:21:08.077 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:21:08.077 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:21:08.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:21:08.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:21:08.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:21:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:21:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:21:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:21:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:21:08.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:21:08.161 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:08.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:21:08.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-04-01T05:21:08.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-04-01T05:21:08.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-04-01T05:21:08.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:08.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:21:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299394 2024-04-01T05:21:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299394 2024-04-01T05:21:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-223338299394' 2024-04-01T05:21:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:08.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:21:08.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-04-01T05:21:08.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-04-01T05:21:08.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-223338299394 2-64424509467' 2024-04-01T05:21:08.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:08.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:21:08.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495621 2024-04-01T05:21:08.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495621 2024-04-01T05:21:08.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-223338299394 2-64424509467 3-197568495621' 2024-04-01T05:21:08.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:08.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-04-01T05:21:08.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:08.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:21:08.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-04-01T05:21:08.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:08.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-04-01T05:21:08.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-01T05:21:08.969 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836516 2024-04-01T05:21:08.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:09.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836516 2024-04-01T05:21:09.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:21:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:21:10.280 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:10.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836516 2024-04-01T05:21:10.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:10.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299394 2024-04-01T05:21:10.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:10.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:21:10.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299394 2024-04-01T05:21:10.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299394 2024-04-01T05:21:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299394' 2024-04-01T05:21:10.576 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 223338299394 2024-04-01T05:21:10.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:21:10.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299394 -lt 223338299394 2024-04-01T05:21:10.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:10.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-04-01T05:21:10.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:10.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:21:10.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-04-01T05:21:10.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:10.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-04-01T05:21:10.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-04-01T05:21:10.861 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509467 2024-04-01T05:21:10.862 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:21:11.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-04-01T05:21:11.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:11.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495621 2024-04-01T05:21:11.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:11.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:21:11.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495621 2024-04-01T05:21:11.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:11.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495621 2024-04-01T05:21:11.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495621' 2024-04-01T05:21:11.153 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 197568495621 2024-04-01T05:21:11.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:21:11.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495622 -lt 197568495621 2024-04-01T05:21:11.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:21:11.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:11.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:11.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:21:11.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:11.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:11.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:11.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:11.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:11.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:11.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:12.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:21:12.073 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:12.073 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:12.074 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:21:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:21:12.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:21:12.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:21:12.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:21:12.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:21:12.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:21:12.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:21:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:21:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:21:12.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:21:12.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:21:12.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:12.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:12.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:12.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:20:43.762877+0000 2024-04-01T05:21:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:21:13.263 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:20:43.762877+0000 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:20:43.762877+0000 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:13.275 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:13.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:13.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:13.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:13.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:43.762877+0000 '>' 2024-04-01T05:20:43.762877+0000 2024-04-01T05:21:13.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:14.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:14.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:14.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:14.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:14.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:14.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:14.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:14.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:20:43.762877+0000 '>' 2024-04-01T05:20:43.762877+0000 2024-04-01T05:21:14.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:15.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:15.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:15.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:15.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:15.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:15.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:15.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:13.962161+0000 '>' 2024-04-01T05:20:43.762877+0000 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:21:16.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:21:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:21:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:21:18.035 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:21:18.035 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:21:18.035 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:21:18.035 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:21:18.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:21:18.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:21:18.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:21:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:21:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:21:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:21:18.306 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:21:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:21:18.307 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:21:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:21:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:21:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:21:18.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:21:18.333 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:18.333+0000 7f1d348b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:18.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:18.335+0000 7f1d348b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:18.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:18.336+0000 7f1d348b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:18.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:21:18.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:21:18.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:18.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:21:18.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:18.862+0000 7f1d348b1640 -1 Falling back to public interface 2024-04-01T05:21:18.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:19.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:19.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:19.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:21:19.884 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:21:19.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:19.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:21:20.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:21.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:21:21.080+0000 7f1d348b1640 -1 osd.3 54 log_to_monitors true 2024-04-01T05:21:21.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:21.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:21.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:21:21.192 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:21:21.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:21.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:21:21.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:22.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:22.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:22.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:21:22.498 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:21:22.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:22.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:21:22.793 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 55 last_clean_interval [46,54) [v2:127.0.0.1:6826/1015268280,v1:127.0.0.1:6827/1015268280] [v2:127.0.0.1:6828/1015268280,v1:127.0.0.1:6829/1015268280] exists,up 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:21:22.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:21:22.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:21:22.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:21:22.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:21:22.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:21:22.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:21:22.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:21:22.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:21:22.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:21:22.878 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:21:23.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:21:23.255 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:21:23.255 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:21:23.255 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:21:23.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:21:23.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:23.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:21:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-04-01T05:21:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-04-01T05:21:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-04-01T05:21:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:23.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:21:23.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299398 2024-04-01T05:21:23.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299398 2024-04-01T05:21:23.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-223338299398' 2024-04-01T05:21:23.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:23.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:21:23.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-04-01T05:21:23.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-04-01T05:21:23.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-223338299398 2-64424509471' 2024-04-01T05:21:23.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:23.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:21:23.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-04-01T05:21:23.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-04-01T05:21:23.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-223338299398 2-64424509471 3-253403070466' 2024-04-01T05:21:23.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:23.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-04-01T05:21:23.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:23.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:21:23.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-04-01T05:21:23.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:23.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-04-01T05:21:23.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-04-01T05:21:23.703 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836520 2024-04-01T05:21:23.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:23.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-04-01T05:21:23.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:21:24.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:21:24.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:25.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-04-01T05:21:25.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:25.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299398 2024-04-01T05:21:25.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:25.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:21:25.290 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299398 2024-04-01T05:21:25.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:25.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299398 2024-04-01T05:21:25.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299398' 2024-04-01T05:21:25.292 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 223338299398 2024-04-01T05:21:25.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:21:25.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299398 -lt 223338299398 2024-04-01T05:21:25.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:25.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-04-01T05:21:25.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:25.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:21:25.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-04-01T05:21:25.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:25.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-04-01T05:21:25.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-01T05:21:25.620 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509471 2024-04-01T05:21:25.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:21:25.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-04-01T05:21:25.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:25.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-04-01T05:21:25.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:25.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:21:25.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-04-01T05:21:25.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:25.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-04-01T05:21:25.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-04-01T05:21:25.905 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 253403070466 2024-04-01T05:21:25.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:21:26.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-04-01T05:21:26.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:21:26.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:26.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:26.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:21:26.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:26.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:26.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:26.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:26.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:26.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:26.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:26.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:21:26.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:26.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:26.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:27.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:21:27.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:21:27.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:21:27.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:21:27.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:21:27.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 166353"' 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 166353' 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:21:27.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 166355"' 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 166355' 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 166353 166355' 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 166353 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/166356: /' 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:21:27.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/166358: /' 2024-04-01T05:21:29.987 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:21:29.987 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:21:29.987 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:21:29.987 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:21:29.987 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:29.987 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:21:29.988 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:21:29.989 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:21:29.990 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:21:29.991 INFO:tasks.workunit.client.0.smithi067.stderr:166356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: start osd.1 2024-04-01T05:21:29.992 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:21:30.033 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: remove 2#2:eb822e21:::SOMETHING:head# 2024-04-01T05:21:30.034 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:21:30.035 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:30.036 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:21:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:21:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:21:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:21:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:21:30.037 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: start osd.2 2024-04-01T05:21:30.038 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:21:34.656 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:21:34.656 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:21:34.656 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2024-04-01T05:21:30.020+0000 7f8487215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2024-04-01T05:21:30.022+0000 7f8487215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2024-04-01T05:21:30.023+0000 7f8487215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:21:34.657 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 0 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2024-04-01T05:21:30.538+0000 7f8487215640 -1 Falling back to public interface 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 1 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:34.658 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2024-04-01T05:21:33.198+0000 7f8487215640 -1 osd.1 60 log_to_monitors true 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 3 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166356: osd.1 up in weight 1 up_from 66 up_thru 55 down_at 61 last_166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:21:34.659 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2024-04-01T05:21:30.063+0000 7efe87468640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2024-04-01T05:21:30.065+0000 7efe87468640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2024-04-01T05:21:30.067+0000 7efe87468640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 0 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.660 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2024-04-01T05:21:30.584+0000 7efe87468640 -1 Falling back to public interface 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 1 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.661 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2024-04-01T05:21:33.495+0000 7efe87468640 -1 osd.2 60 log_to_monitors true 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 3 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:34.662 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:166358: osd.2 up in weight 1 up_from 67 up_thru 49 down_at 61 last_clean_interval [15,60) [v2:127.0.0.1:6818/704386215,v1:127.0.0.1:6819/704386215] [v2:127.0.0.1:6820/704386215,v1:127.0.0.1:6821/704386215] exists,up d0be321f-1cd9-41dc-a51d-5e81df352119 2024-04-01T05:21:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:21:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:21:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:21:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:21:35.409 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 1 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 2 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: 3' 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-04-01T05:21:35.410 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-04-01T05:21:35.411 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-283467841538' 2024-04-01T05:21:35.411 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tellclean_interval [52,60) [v2:127.0.0.1:6810/4188751447,v1:127.0.0.1:6811/4188751447] [v2:127.0.0.1:6812/4188751447,v1:127.0.0.1:6813/4188751447] exists,up 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:21:35.593 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 1 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 2 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: 3' 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:21:35.594 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-283467841539' 2024-04-01T05:21:35.595 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:36.628 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:21:36.628 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-283467841539 2-287762808835' 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-283467841539 2-287762808835 3-253403070470' 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-04-01T05:21:36.629 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: waiting osd.0 seq 21474836525 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841539 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841539 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841539' 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: waiting osd.1 seq 283467841539 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:21:36.630 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-04-01T05:21:36.631 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:36.631 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808835 2024-04-01T05:21:36.631 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:36.631 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:21:36.631 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808835 2024-04-01T05:21:37.804 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/ceph osd.2 flush_pg_stats 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-283467841538 2-287762808834' 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070469 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070469 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-283467841538 2-287762808834 3-253403070469' 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:21:37.805 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: waiting osd.0 seq 21474836524 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836524 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:37.806 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: waiting osd.1 seq 283467841538 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-04-01T05:21:37.807 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166358: test/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: waiting osd.2 seq 287762808835 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: waiting osd.3 seq 253403070470 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070470 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:38.239 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:21:38.240 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:21:39.193 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalon//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-287762808834 2024-04-01T05:21:39.193 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:39.193 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:21:39.193 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-287762808834 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: waiting osd.2 seq 287762808834 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070469 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070469 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070469 2024-04-01T05:21:39.194 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070469' 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: waiting osd.3 seq 253403070469 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070469 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:39.195 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:39.196 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:21:39.196 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:39.556 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.cliene/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:39.557 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:21:39.558 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:21:39.559 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:21:40.653 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1t.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:40.654 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:21:40.655 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:21:40.656 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clon653: wait_for_clean: true 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:41.023 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:21:41.024 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:41.025 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:41.515 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='e.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:21:41.515 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:21:41.516 INFO:tasks.workunit.client.0.smithi067.stderr:166358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:21:41.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:21:41.737 INFO:tasks.workunit.client.0.smithi067.stderr:select(contains("stale") | not)' 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:166356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 166355 2024-04-01T05:21:41.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:21:41.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:21:42.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:42.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:42.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:21:42.613 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:42.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:42.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:13.962161+0000 '>' 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:42.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:43.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:43.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:43.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:43.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:43.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:43.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:43.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:44.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:13.962161+0000 '>' 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:44.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:45.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:45.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:45.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:45.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:45.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:45.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:45.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:45.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:13.962161+0000 '>' 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:45.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:46.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:46.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:46.495 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:46.495 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:46.495 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:46.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:46.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:46.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:13.962161+0000 '>' 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:46.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:47.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:47.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:47.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:47.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:47.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:47.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:47.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:13.962161+0000 '>' 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:48.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:21:49.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:21:49.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:21:49.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:21:49.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:21:49.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:21:49.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:21:49.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:43.017219+0000 '>' 2024-04-01T05:21:13.962161+0000 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 169524"' 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 169524' 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:21:49.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 169526"' 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 169526' 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 169524 169526' 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/169527: /' 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 169524 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:21:49.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/169529: /' 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.817 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:21:51.818 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: _ 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: hinfo_key 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: snapset 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:21:51.819 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:21:51.820 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:21:51.821 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:21:51.822 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:21:51.822 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:21:51.822 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:21:51.822 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:21:51.822 INFO:tasks.workunit.client.0.smithi067.stderr:169527: start osd.1 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:21:51.866 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: _ 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: hinfo_key 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: snapset 2024-04-01T05:21:51.867 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:21:51.868 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:21:51.869 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:21:51.870 INFO:tasks.workunit.client.0.smithi067.stderr:169529: start osd.2 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2024-04-01T05:21:51.849+0000 7f62f9ec8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2024-04-01T05:21:51.852+0000 7f62f9ec8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2024-04-01T05:21:51.853+0000 7f62f9ec8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:21:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 0 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2024-04-01T05:21:52.365+0000 7f62f9ec8640 -1 Falling back to public interface 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:21:56.405 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 1 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2024-04-01T05:21:54.941+0000 7f62f9ec8640 -1 osd.1 70 log_to_monitors true 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:21:56.406 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 3 2024-04-01T05:21:56.407 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.407 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169527: osd.1 up in weight 1 up_from 76 up_thru 68 down_aovery_ops 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2024-04-01T05:21:51.896+0000 7f2086664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2024-04-01T05:21:51.899+0000 7f2086664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2024-04-01T05:21:51.900+0000 7f2086664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:21:56.409 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 0 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2024-04-01T05:21:52.412+0000 7f2086664640 -1 Falling back to public interface 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.410 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 1 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2024-04-01T05:21:55.017+0000 7f2086664640 -1 osd.2 70 log_to_monitors true 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:21:56.411 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:21:56.412 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 3 2024-04-01T05:21:56.412 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:21:56.412 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: osd.2 up in weight 1 up_from 76 up_thru 49 down_at 71 last_clean_interval [67,70) [v2:127.0.0.1:6818/2278111160,v1:127.0.0.1:6819/2278111160] [v2:127.0.0.1:6820/2278111160,v1:127.0.0.1:6821/2278111160] exists,up d0be321f-1cd9-41dc-a51d-5e81df352119 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:21:57.221 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:21:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 1 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 2 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: 3' 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-04-01T05:21:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-326417514498' 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 71 last_clean_interval [66,70) [v2:127.0.0.1:6810/3990070778,v1:127.0.0.1:6811/3990070778] [v2:127.0.0.1:6812/3990070778,v1:127.0.0.1:6813/3990070778] exists,up 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:21:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 1 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 2 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: 3' 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-04-01T05:21:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499' 2024-04-01T05:21:57.226 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:59.368 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:21:59.368 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-04-01T05:21:59.368 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-04-01T05:21:59.368 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-326417514498 2-326417514499' 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070477 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070477 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-326417514498 2-326417514499 3-253403070477' 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-04-01T05:21:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: waiting osd.0 seq 21474836531 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836531 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514498 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514498 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169529: waiting osd.1 seq 326417514498 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499 2-326417514498' 2024-04-01T05:21:59.371 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499 2-326417514498 3-253403070476' 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: waiting osd.0 seq 21474836530 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:59.372 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836530 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514499 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514499 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: waiting osd.1 seq 326417514499 2024-04-01T05:21:59.373 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:21:59.374 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514499 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514499 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514499' 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: waiting osd.2 seq 326417514499 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:22:00.642 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070477 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070477 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070477 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070477' 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: waiting osd.3 seq 253403070477 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070477 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:00.643 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:22:00.644 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephte$seqs 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514498 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514498 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-04-01T05:22:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:169527: waiting osd.2 seq 326417514498 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: waiting osd.3 seq 253403070476 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070477 -lt 253403070476 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:22:00.647 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:22:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:00.998 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:00.999 INFO:tasks.workunit.client.0.smithi067.stderr:169529: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:00.999 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:22:00.999 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:22:00.999 INFO:tasks.workunit.client.0.smithi067.stderr:169529: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:22:00.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:169527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:169527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 169526 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:22:01.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:22:01.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:22:01.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:22:01.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:22:01.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-04-01T05:22:01.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:22:01.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:22:01.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-04-01T05:22:01.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 172203"' 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 172203' 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 172205"' 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 172205' 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:22:01.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 172203 172205' 2024-04-01T05:22:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:22:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:22:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 172203 2024-04-01T05:22:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/172206: /' 2024-04-01T05:22:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:22:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/172208: /' 2024-04-01T05:22:03.667 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.667 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:22:03.668 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:22:03.669 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:22:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:22:03.670 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:03.671 INFO:tasks.workunit.client.0.smithi067.stderr:172208: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: start osd.1 2024-04-01T05:22:03.672 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:22:03.932 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:22:03.933 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:22:03.934 INFO:tasks.workunit.client.0.smithi067.stderr:172206: remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:22:03.935 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:03.936 INFO:tasks.workunit.client.0.smithi067.stderr:172206: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:22:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:22:03.938 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:22:03.938 INFO:tasks.workunit.client.0.smithi067.stderr:172206: start osd.3 2024-04-01T05:22:03.938 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:22:08.344 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:22:08.344 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:22:08.344 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:22:08.344 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:22:08.344 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:22:08.344 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2024-04-01T05:22:03.699+0000 7f0e20613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2024-04-01T05:22:03.701+0000 7f0e20613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2024-04-01T05:22:03.703+0000 7f0e20613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 0 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2024-04-01T05:22:04.223+0000 7f0e20613640 -1 Falling back to public interface 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:08.345 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 1 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:08.346 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2024-04-01T05:22:06.965+0000 7f0e20613640 -1 osd.1 78 log_to_monitors true 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 3 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.347 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172208: osd.1 up in weight 1 up_from 83 up_thru 83 down_at 79 last_172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2024-04-01T05:22:03.961+0000 7f24da8ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:08.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2024-04-01T05:22:03.963+0000 7f24da8ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2024-04-01T05:22:03.964+0000 7f24da8ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 0 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2024-04-01T05:22:04.479+0000 7f24da8ac640 -1 Falling back to public interface 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:08.547 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 1 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2024-04-01T05:22:07.155+0000 7f24da8ac640 -1 osd.3 78 log_to_monitors true 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:08.548 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:08.549 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:22:08.549 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 3 2024-04-01T05:22:08.549 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:08.549 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:09.190 INFO:tasks.workunit.client.0.smithi067.stderr:172206: osd.3 up in weight 1 up_from 84 up_thru 84 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6810/1248258147,v1:127.0.0.1:6811/1248258147] [v2:127.0.0.1:6812/1248258147,v1:127.0.0.1:6813/1248258147] exists,up 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:22:09.190 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:22:09.191 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 1 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 2 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: 3' 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:09.192 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:22:09.193 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-04-01T05:22:09.193 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-04-01T05:22:09.193 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570' 2024-04-01T05:22:09.193 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:09.303 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [59,78) [v2:127.0.0.1:6826/1722572029,v1:127.0.0.1:6827/1722572029] [v2:127.0.0.1:6828/1722572029,v1:127.0.0.1:6829/1722572029] exists,up 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:22:09.303 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:22:09.303 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:22:09.304 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 1 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 2 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: 3' 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:09.305 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:22:09.306 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-04-01T05:22:09.306 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-04-01T05:22:09.306 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-356482285571' 2024-04-01T05:22:09.306 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514502 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514502 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570 2-326417514502' 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-04-01T05:22:11.399 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570 2-326417514502 3-360777252866' 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: waiting osd.0 seq 21474836534 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:22:11.400 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836534 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: waiting osd.1 seq 356482285570 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285570 2024-04-01T05:22:11.401 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:11.542 INFO:tasks.workunit.client.0.smithi067.stderr:1722tell osd.2 flush_pg_stats 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514503 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514503 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-356482285571 2-326417514503' 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-356482285571 2-326417514503 3-360777252867' 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:22:11.543 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-04-01T05:22:11.544 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:11.544 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-04-01T05:22:11.544 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-04-01T05:22:11.545 INFO:tasks.workunit.client.0.smithi067.stderr:172206: waiting osd.0 seq 21474836535 2024-04-01T05:22:11.545 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:11.545 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836535 2024-04-01T05:22:11.545 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:22:11.545 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:22:11.545 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836535 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285571 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285571 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: waiting osd.1 seq 356482285571 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285571 2024-04-01T05:22:11.546 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:12.685 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514502 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514502 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514502 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514502' 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: waiting osd.2 seq 326417514502 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514503 -lt 326417514502 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-360777252866 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:12.686 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-360777252866 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 360777252866' 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: waiting osd.3 seq 360777252866 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:12.687 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:12.688 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:12.688 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:12.688 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:22:12.688 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:12.773 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.c06: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514503 2024-04-01T05:22:12.773 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514503 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514503 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514503' 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: waiting osd.2 seq 326417514503 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514503 -lt 326417514503 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-360777252867 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-360777252867 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 360777252867' 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: waiting osd.3 seq 360777252867 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252867 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:22:12.775 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:22:12.776 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:172208: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:22:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:22:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:172208: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:22:13.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:172206: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:172206: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 172205 2024-04-01T05:22:13.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:22:13.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:22:13.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:22:13.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:13.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:22:14.005 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:22:14.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:22:14.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:43.017219+0000 '>' 2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:14.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:22:15.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:22:15.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:22:15.341 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:22:15.341 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:22:15.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:22:15.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:22:15.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:22:15.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:43.017219+0000 '>' 2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:15.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:22:16.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:22:16.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:22:16.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:22:16.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:22:16.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:22:16.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:22:16.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:22:16.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:21:43.017219+0000 '>' 2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:16.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:22:17.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:22:17.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:22:17.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:22:17.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:22:17.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:22:17.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:22:17.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:22:14.066026+0000 '>' 2024-04-01T05:21:43.017219+0000 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 175030"' 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 175030' 2024-04-01T05:22:18.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 175032"' 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 175032' 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 175030 175032' 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 175030 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/175033: /' 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:22:18.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/175035: /' 2024-04-01T05:22:20.723 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:22:20.724 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: _ 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: hinfo_key 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: snapset 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:22:20.725 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:22:20.726 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:22:20.726 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:22:20.726 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:22:20.726 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:22:20.726 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:22:20.726 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:22:20.728 INFO:tasks.workunit.client.0.smithi067.stderr:175033: start osd.3 2024-04-01T05:22:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:22:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:22:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: _ 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: hinfo_key 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: snapset 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:22:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:22:21.085 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:22:21.086 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:22:21.087 INFO:tasks.workunit.client.0.smithi067.stderr:175035: start osd.1 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:22:24.952 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2024-04-01T05:22:20.756+0000 7f500cc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2024-04-01T05:22:20.758+0000 7f500cc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2024-04-01T05:22:20.760+0000 7f500cc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 0 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2024-04-01T05:22:21.278+0000 7f500cc13640 -1 Falling back to public interface 2024-04-01T05:22:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 1 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2024-04-01T05:22:23.623+0000 7f500cc13640 -1 osd.3 87 log_to_monitors true 2024-04-01T05:22:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:22:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 3 2024-04-01T05:22:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-04-01T05:22:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:22:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2024-04-01T05:22:21.113+0000 7f28adb8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2024-04-01T05:22:21.154+0000 7f28adb8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2024-04-01T05:22:21.155+0000 7f28adb8f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:22:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:22:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:22:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:22:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:22:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 0 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2024-04-01T05:22:21.667+0000 7f28adb8f640 -1 Falling back to public interface 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:25.695 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 1 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:25.696 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2024-04-01T05:22:24.327+0000 7f28adb8f640 -1 osd.1 86 log_to_monitors true 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:25.697 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:25.698 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:22:25.698 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 3 2024-04-01T05:22:25.698 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:25.698 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:22:26.151 INFO:tasks.workunit.client.0.smithi067.stderr:175035: osd.1 up in weight 1 up_from 93 up_thru 93 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:22:26.151 INFO:tasks.workunit.client.0.smithi067.stderr:175033: osd.3 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [84,87) [v2:127.0.0.1:6810/2227843542,v1:127.0.0.1:6811/2227843542] [v2:127.0.0.1:6812/2227843542,v1:127.0.0.1:6813/2227843542] exists,up 23c8b24d-b8b4-4ed0-b072-b5f9f063108d 2024-04-01T05:22:26.151 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:22:26.151 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:22:26.151 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:22:26.152 INFO:tasks.workunit.client.0.smithi067.stderr:175033: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:22:26.153 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 1 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 2 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: 3' 2024-04-01T05:22:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:22:26.155 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:26.155 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:22:26.155 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-04-01T05:22:26.155 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-04-01T05:22:26.155 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-04-01T05:22:26.155 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:26.156 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:22:26.156 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-04-01T05:22:26.156 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-04-01T05:22:26.156 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-399431958530' 2024-04-01T05:22:26.506 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 87 last_clean_interval [83,86) [v2:127.0.0.1:6826/4130225893,v1:127.0.0.1:6827/4130225893] [v2:127.0.0.1:6828/4130225893,v1:127.0.0.1:6829/4130225893] exists,up 18bd570f-9be1-4dc3-9b20-bc20cffbb88e 2024-04-01T05:22:26.506 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:22:26.506 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:22:26.506 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:22:26.506 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:22:26.506 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:22:26.507 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 1 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 2 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: 3' 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-04-01T05:22:26.508 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-399431958531' 2024-04-01T05:22:26.509 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stan $ids 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514508 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514508 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-399431958530 2-326417514508' 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-399431958530 2-326417514508 3-390842023938' 2024-04-01T05:22:28.461 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: waiting osd.0 seq 21474836540 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836540 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958530 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:22:28.462 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958530 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958530' 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: waiting osd.1 seq 399431958530 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:22:28.463 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/uts: ceph tell osd.2 flush_pg_stats 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514509 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514509 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-399431958531 2-326417514509' 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023939 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023939 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-399431958531 2-326417514509 3-390842023939' 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-04-01T05:22:28.732 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: waiting osd.0 seq 21474836541 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836541 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958531 2024-04-01T05:22:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958531 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958531' 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: waiting osd.1 seq 399431958531 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:22:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-04-01T05:22:29.765 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:29.765 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514508 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514508 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514508 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514508' 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: waiting osd.2 seq 326417514508 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514509 -lt 326417514508 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023938 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:29.766 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023938 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023938' 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: waiting osd.3 seq 390842023938 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023938 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:29.767 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:29.768 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:29.768 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:29.768 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:29.768 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:29.768 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:22:30.010 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/q$seqs 2024-04-01T05:22:30.010 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514509 2024-04-01T05:22:30.010 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:30.010 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:22:30.010 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514509 2024-04-01T05:22:30.010 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514509 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514509' 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: waiting osd.2 seq 326417514509 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514509 -lt 326417514509 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023939 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023939 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023939 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-04-01T05:22:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:175035: waiting osd.3 seq 390842023939 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023939 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:22:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:30.926 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtea/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:30.926 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:30.926 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:30.926 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:30.927 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:175033: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:175033: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:22:30.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 175032 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:22:31.118 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:22:31.119 INFO:tasks.workunit.client.0.smithi067.stderr:175035: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:22:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:22:31.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:22:31.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:22:31.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:22:31.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:22:31.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:22:31.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:22:31.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:22:31.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:22:31.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:22:31.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:22:31.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:22:31.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:22:31.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:22:31.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:22:31.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:22:31.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:22:31.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:22:31.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:31.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:31.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:22:31.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:22:31.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:22:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:22:31.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:22:31.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:22:31.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:22:31.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:22:31.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:22:31.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:22:31.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:22:31.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:22:31.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:22:31.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:22:31.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:22:31.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:22:31.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:22:31.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:22:31.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:22:31.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:31.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:31.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:22:31.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:22:31.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:22:31.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:22:31.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:22:31.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:31.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:31.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:22:31.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:22:31.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:22:31.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:31.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:22:31.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:22:31.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:22:31.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:22:31.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:22:31.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:22:31.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:22:31.990 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:22:31.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:22:31.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:22:32.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:22:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:22:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:32.882 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:32.882 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:32.882 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:32.882 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:32.882 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:22:32.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:22:32.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:22:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:22:32.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d79e1f30-0801-4640-9652-8bcc759003e7 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d79e1f30-0801-4640-9652-8bcc759003e7' 2024-04-01T05:22:32.916 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 d79e1f30-0801-4640-9652-8bcc759003e7 2024-04-01T05:22:32.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:22:32.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYRApmfJqJNxAAuYDSExpnNeEQngq37n1BhA== 2024-04-01T05:22:32.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYRApmfJqJNxAAuYDSExpnNeEQngq37n1BhA=="}' 2024-04-01T05:22:32.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d79e1f30-0801-4640-9652-8bcc759003e7 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:22:33.162 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:22:33.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:22:33.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYRApmfJqJNxAAuYDSExpnNeEQngq37n1BhA== --osd-uuid d79e1f30-0801-4640-9652-8bcc759003e7 2024-04-01T05:22:33.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.202+0000 7f36a5a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:33.213 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.214+0000 7f36a5a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.215+0000 7f36a5a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.215+0000 7f36a5a14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:22:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.215+0000 7f36a5a14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:22:33.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.215+0000 7f36a5a14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:22:33.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:33.306+0000 7f36a5a14640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:22:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:22:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:22:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:22:38.823 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:22:38.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:22:39.190 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:22:39.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:22:39.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:22:39.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:22:39.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:22:39.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:22:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:39.217+0000 7fe9a7591640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:39.246 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:39.246+0000 7fe9a7591640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:39.247 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:39.247+0000 7fe9a7591640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:22:39.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:22:39.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:22:39.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:22:39.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:22:39.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:22:39.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:39.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:22:39.475 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:22:39.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:39.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:39.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:39.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:39.771+0000 7fe9a7591640 -1 Falling back to public interface 2024-04-01T05:22:40.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:40.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:40.762 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:22:40.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:22:40.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:40.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:41.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:42.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:42.057 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:22:42.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:22:42.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:42.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:42.103 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:22:42.103+0000 7fe9a7591640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:22:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:43.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:43.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:43.356 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:22:43.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:22:43.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:43.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:43.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:44.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:44.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:44.650 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:22:44.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:22:44.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:44.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:45.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:46.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:46.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:46.043 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:22:46.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:22:46.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:46.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:46.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:47.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:47.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:47.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:22:47.338 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:22:47.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:47.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:47.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:48.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:48.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:48.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:22:48.666 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:22:48.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:48.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:49.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:50.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:50.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:50.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:22:50.067 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:22:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:50.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:50.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:51.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:51.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:51.362 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:22:51.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:22:51.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:51.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:51.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:52.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:52.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:52.656 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:22:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:22:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:52.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:53.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:54.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:54.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:54.364 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:22:54.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:22:54.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:54.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:54.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:55.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:55.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:55.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:22:55.955 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:22:55.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:55.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:56.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:57.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:57.405 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:22:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:22:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:57.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:58.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:22:59.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:22:59.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:22:59.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:22:59.163 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:22:59.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:22:59.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:22:59.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:00.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:00.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:00.587 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:23:00.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:23:00.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:00.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:23:01.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:02.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:02.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:02.185 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:23:02.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:23:02.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:02.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:23:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:03.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:03.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:03.630 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:23:03.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:23:03.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:03.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:23:03.920 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3887784371,v1:127.0.0.1:6803/3887784371] [v2:127.0.0.1:6804/3887784371,v1:127.0.0.1:6805/3887784371] exists,up d79e1f30-0801-4640-9652-8bcc759003e7 2024-04-01T05:23:03.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:23:03.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:23:03.926 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:23:03.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0b7d1c55-e678-477c-873f-c18edafa5932' 2024-04-01T05:23:03.928 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:23:03.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:23:03.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3RApmJUAiOBAA8/avMEdTsaH/89I4xmnUTA== 2024-04-01T05:23:03.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3RApmJUAiOBAA8/avMEdTsaH/89I4xmnUTA=="}' 2024-04-01T05:23:03.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0b7d1c55-e678-477c-873f-c18edafa5932 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:23:04.313 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:23:04.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:23:04.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3RApmJUAiOBAA8/avMEdTsaH/89I4xmnUTA== --osd-uuid 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:23:04.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.352+0000 7f2c25612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:04.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.397+0000 7f2c25612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:04.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.398+0000 7f2c25612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:04.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.399+0000 7f2c25612640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:23:04.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.399+0000 7f2c25612640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:23:04.399 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.399+0000 7f2c25612640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:23:04.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:04.472+0000 7f2c25612640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:23:10.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:23:10.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:23:10.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:23:10.136 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:23:10.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:23:10.509 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:23:10.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:23:10.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:23:10.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:23:10.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:23:10.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:23:10.535 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:10.536+0000 7ff6d1c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:10.537 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:10.538+0000 7ff6d1c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:10.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:10.539+0000 7ff6d1c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:23:10.796 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:23:10.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:10.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:11.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:11.066+0000 7ff6d1c1b640 -1 Falling back to public interface 2024-04-01T05:23:11.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:12.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:12.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:12.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:23:12.107 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:23:12.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:12.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:12.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:13.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:13.370+0000 7ff6d1c1b640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:23:13.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:13.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:13.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:23:13.401 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:23:13.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:13.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:13.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:14.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:14.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:14.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:23:14.805 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:23:14.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:14.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:15.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:16.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:16.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:16.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:23:16.101 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:23:16.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:16.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:16.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:17.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:17.425 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:23:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:23:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:17.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:17.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:18.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:18.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:18.719 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:23:18.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:23:18.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:18.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:19.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:20.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:20.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:20.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:23:20.144 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:23:20.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:20.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:20.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:21.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:21.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:21.446 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:23:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:23:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:21.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:21.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:22.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:22.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:22.743 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:23:22.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:23:22.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:22.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:23.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:24.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:24.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:24.110 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:23:24.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:23:24.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:24.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:24.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:25.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:23:25.408 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:23:25.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:25.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:26.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:27.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:27.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:27.094 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:23:27.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:23:27.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:27.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:27.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:28.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:28.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:28.772 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:23:28.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:23:28.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:28.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:29.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:30.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:30.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:30.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:23:30.508 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:23:30.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:30.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:31.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:32.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:32.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:32.193 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:23:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:23:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:32.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:32.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:33.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:33.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:33.787 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:23:33.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:23:33.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:33.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:34.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:35.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:35.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:35.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:23:35.474 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:23:35.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:35.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:23:35.786 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4085624497,v1:127.0.0.1:6811/4085624497] [v2:127.0.0.1:6812/4085624497,v1:127.0.0.1:6813/4085624497] exists,up 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:23:35.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:23:35.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:23:35.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fa394218-fd37-4c9a-b739-88b5b9399a3d 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fa394218-fd37-4c9a-b739-88b5b9399a3d' 2024-04-01T05:23:35.793 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 fa394218-fd37-4c9a-b739-88b5b9399a3d 2024-04-01T05:23:35.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:23:35.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXRApmcy4qMBAArneStiQklVKPkattklQ2Vw== 2024-04-01T05:23:35.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXRApmcy4qMBAArneStiQklVKPkattklQ2Vw=="}' 2024-04-01T05:23:35.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fa394218-fd37-4c9a-b739-88b5b9399a3d -i td/osd-scrub-repair/2/new.json 2024-04-01T05:23:36.151 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:23:36.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:23:36.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXRApmcy4qMBAArneStiQklVKPkattklQ2Vw== --osd-uuid fa394218-fd37-4c9a-b739-88b5b9399a3d 2024-04-01T05:23:36.190 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.190+0000 7f11a51fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:36.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.227+0000 7f11a51fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:36.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.228+0000 7f11a51fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:36.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.228+0000 7f11a51fd640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:23:36.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.228+0000 7f11a51fd640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:23:36.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.228+0000 7f11a51fd640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:23:36.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:36.319+0000 7f11a51fd640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:23:42.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:23:42.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:23:42.066 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:23:42.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:23:42.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:23:42.434 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:23:42.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:23:42.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:23:42.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:23:42.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:23:42.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:23:42.461 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:42.461+0000 7f98aae0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:42.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:42.463+0000 7f98aae0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:42.464 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:42.464+0000 7f98aae0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:23:42.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:23:42.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:23:42.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:23:42.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:23:42.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:23:42.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:42.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:23:42.719 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:23:42.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:42.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:42.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:42.988+0000 7f98aae0f640 -1 Falling back to public interface 2024-04-01T05:23:43.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:44.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:44.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:44.002 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:23:44.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:23:44.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:44.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:44.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:45.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:45.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:45.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:23:45.294 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:23:45.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:45.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:45.410 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:23:45.410+0000 7f98aae0f640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:23:45.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:46.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:46.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:46.591 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:23:46.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:23:46.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:46.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:46.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:47.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:47.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:47.889 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:23:47.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:23:47.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:47.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:48.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:49.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:49.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:49.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:23:49.181 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:23:49.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:49.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:49.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:50.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:50.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:50.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:23:50.477 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:23:50.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:50.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:50.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:51.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:51.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:51.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:23:51.772 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:23:51.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:51.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:52.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:53.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:53.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:53.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:23:53.158 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:23:53.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:53.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:53.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:54.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:54.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:54.453 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:23:54.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:23:54.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:54.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:54.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:55.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:55.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:55.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:23:55.749 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:23:55.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:55.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:56.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:57.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:57.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:57.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:23:57.163 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:23:57.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:57.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:58.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:23:59.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:23:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:23:59.207 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:23:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:23:59.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:23:59.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:23:59.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:00.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:00.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:00.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:24:00.918 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:24:00.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:00.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:24:01.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:24:02.387 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:24:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:24:03.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:04.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:04.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:04.086 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:24:04.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:24:04.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:04.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:24:04.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:05.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:05.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:05.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:24:05.786 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:24:05.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:05.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:24:06.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:07.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:07.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:07.562 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:24:07.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:24:07.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:07.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:24:07.893 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2082113722,v1:127.0.0.1:6819/2082113722] [v2:127.0.0.1:6820/2082113722,v1:127.0.0.1:6821/2082113722] exists,up fa394218-fd37-4c9a-b739-88b5b9399a3d 2024-04-01T05:24:07.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:24:07.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:24:07.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:24:07.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:24:07.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 53d5d26f-3746-48be-b55b-aae15a84b6cf' 2024-04-01T05:24:07.901 INFO:tasks.workunit.client.0.smithi067.stdout:add osd3 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:24:07.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:24:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3RApmIgx+NhAAzEkKi4J7KWHFrIsd8b1V6A== 2024-04-01T05:24:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3RApmIgx+NhAAzEkKi4J7KWHFrIsd8b1V6A=="}' 2024-04-01T05:24:07.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53d5d26f-3746-48be-b55b-aae15a84b6cf -i td/osd-scrub-repair/3/new.json 2024-04-01T05:24:08.280 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:24:08.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-04-01T05:24:08.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3RApmIgx+NhAAzEkKi4J7KWHFrIsd8b1V6A== --osd-uuid 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:24:08.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.318+0000 7fe38260f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:24:08.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.355+0000 7fe38260f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:24:08.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.356+0000 7fe38260f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:24:08.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.356+0000 7fe38260f640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:24:08.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.356+0000 7fe38260f640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:24:08.357 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.356+0000 7fe38260f640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:24:08.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:08.430+0000 7fe38260f640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-04-01T05:24:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-04-01T05:24:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:24:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-01T05:24:14.670 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd3 key to auth repository 2024-04-01T05:24:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:24:15.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-01T05:24:15.042 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.3 2024-04-01T05:24:15.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:24:15.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:24:15.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:24:15.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:24:15.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:15.067+0000 7fbbe86b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:24:15.069 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:15.069+0000 7fbbe86b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:24:15.070 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:15.070+0000 7fbbe86b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:24:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-01T05:24:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:24:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:24:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:24:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:24:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:24:15.328 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:24:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:15.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:15.592+0000 7fbbe86b5640 -1 Falling back to public interface 2024-04-01T05:24:15.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:16.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:16.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:16.611 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:24:16.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:24:16.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:16.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:16.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:17.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:17.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:17.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:24:17.904 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:24:17.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:17.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:18.011 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:24:18.010+0000 7fbbe86b5640 -1 osd.3 0 log_to_monitors true 2024-04-01T05:24:18.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:19.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:19.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:19.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:24:19.271 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:24:19.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:19.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:20.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:20.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:20.568 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:24:20.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:24:20.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:20.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:20.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:21.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:21.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:21.889 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:24:21.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:24:21.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:21.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:22.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:23.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:23.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:23.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:24:23.263 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:24:23.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:23.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:23.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:24.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:24.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:24.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:24:24.554 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:24:24.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:24.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:24.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:25.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:25.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:25.873 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:24:25.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:24:25.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:25.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:26.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:27.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:27.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:27.238 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:24:27.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:24:27.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:27.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:27.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:28.532 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:24:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:24:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:28.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:28.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:29.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:29.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:29.848 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:24:29.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:24:29.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:29.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:30.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:31.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:31.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:31.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:24:31.142 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:24:31.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:31.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:31.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:32.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:32.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:32.478 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:24:32.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:24:32.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:32.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:33.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:34.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:34.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:34.165 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:24:34.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:24:34.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:34.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:34.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:35.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:35.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:35.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:24:35.681 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:24:35.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:35.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:36.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:37.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:37.400 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:24:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:24:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:37.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:38.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:39.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:39.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:39.240 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:24:39.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:24:39.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:39.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:39.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:24:40.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:24:40.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:24:40.935 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:24:40.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:24:40.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:24:40.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:24:41.273 INFO:tasks.workunit.client.0.smithi067.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2629022911,v1:127.0.0.1:6827/2629022911] [v2:127.0.0.1:6828/2629022911,v1:127.0.0.1:6829/2629022911] exists,up 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:24:41.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:24:41.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:24:41.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:24:41.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-04-01T05:24:41.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:24:41.615 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:24:41.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:24:41.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:24:42.006 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:24:42.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:24:43.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:24:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-04-01T05:24:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:24:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:24:45.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:24:45.097 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:24:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:24:45.098 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:24:45.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:24:45.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:24:45.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:24:45.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:24:45.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:24:45.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:24:45.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:24:45.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:24:45.193 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:24:45.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:24:45.478 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:24:45.478 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:24:45.478 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:24:45.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:24:45.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:45.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:24:45.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-04-01T05:24:45.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-04-01T05:24:45.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-04-01T05:24:45.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:45.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:24:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-04-01T05:24:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-04-01T05:24:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979' 2024-04-01T05:24:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:45.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:24:45.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-04-01T05:24:45.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-04-01T05:24:45.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979 2-64424509454' 2024-04-01T05:24:45.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:45.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:24:45.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-04-01T05:24:45.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-04-01T05:24:45.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672979 2-64424509454 3-85899345928' 2024-04-01T05:24:45.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:45.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-04-01T05:24:45.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:45.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:24:45.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-04-01T05:24:45.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:45.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-04-01T05:24:45.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-01T05:24:45.903 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836503 2024-04-01T05:24:45.903 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:24:46.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-04-01T05:24:46.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:24:47.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:24:47.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:24:47.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-04-01T05:24:47.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:47.512 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-04-01T05:24:47.513 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:47.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:24:47.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-04-01T05:24:47.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:47.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-04-01T05:24:47.517 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672979 2024-04-01T05:24:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-01T05:24:47.518 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:24:47.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672979 2024-04-01T05:24:47.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:47.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-04-01T05:24:47.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:47.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:24:47.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-04-01T05:24:47.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:47.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-04-01T05:24:47.802 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509454 2024-04-01T05:24:47.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-01T05:24:47.803 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:24:48.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-01T05:24:48.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:48.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345928 2024-04-01T05:24:48.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:48.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:24:48.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345928 2024-04-01T05:24:48.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:48.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-04-01T05:24:48.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-04-01T05:24:48.127 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345928 2024-04-01T05:24:48.127 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:24:48.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345928 2024-04-01T05:24:48.424 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:24:48.424 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:24:48.424 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:24:48.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:24:49.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:24:49.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:24:49.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:24:49.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:24:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-04-01T05:24:49.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:24:49.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:24:50.596 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:24:50.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:24:51.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-04-01T05:24:51.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-04-01T05:24:52.030 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 2 allow_ec_overwrites to true 2024-04-01T05:24:52.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:24:52.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:24:52.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:24:52.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:24:52.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:24:52.042 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:24:52.043 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:24:52.043 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:24:52.043 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:24:52.043 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:24:52.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:24:52.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:24:52.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:24:52.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:24:52.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:24:52.128 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:24:52.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:24:52.408 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:24:52.408 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:24:52.408 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:24:52.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:24:52.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:52.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:24:52.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-04-01T05:24:52.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-04-01T05:24:52.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-04-01T05:24:52.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:52.528 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:24:52.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-04-01T05:24:52.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-04-01T05:24:52.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672982' 2024-04-01T05:24:52.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:52.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:24:52.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-04-01T05:24:52.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-04-01T05:24:52.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509456' 2024-04-01T05:24:52.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:24:52.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:24:52.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-04-01T05:24:52.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-04-01T05:24:52.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509456 3-85899345931' 2024-04-01T05:24:52.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:52.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-04-01T05:24:52.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:52.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:24:52.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-04-01T05:24:52.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:52.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-04-01T05:24:52.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-01T05:24:52.846 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836505 2024-04-01T05:24:52.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:24:53.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-04-01T05:24:53.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:24:54.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:24:54.128 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:24:54.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-01T05:24:54.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:54.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-04-01T05:24:54.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:54.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:24:54.422 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-04-01T05:24:54.422 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:54.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-04-01T05:24:54.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-01T05:24:54.424 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672982 2024-04-01T05:24:54.424 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:24:54.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-01T05:24:54.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:54.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-04-01T05:24:54.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:54.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:24:54.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-04-01T05:24:54.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:54.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-04-01T05:24:54.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-01T05:24:54.733 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509456 2024-04-01T05:24:54.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:24:55.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-04-01T05:24:55.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:24:55.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-04-01T05:24:55.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:24:55.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:24:55.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-04-01T05:24:55.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:24:55.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-04-01T05:24:55.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-04-01T05:24:55.018 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345931 2024-04-01T05:24:55.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:24:55.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-04-01T05:24:55.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:24:55.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:24:55.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:24:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:24:55.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:24:55.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:24:55.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:24:55.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-04-01T05:24:56.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:24:56.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:24:56.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-04-01T05:24:56.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:24:56.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:24:56.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:24:56.683 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:24:56.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:24:57.094 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:24:57.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:24:57.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:24:57.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:24:57.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-04-01T05:24:57.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:24:57.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:24:57.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:24:57.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:24:57.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-04-01T05:24:57.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-04-01T05:24:57.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T05:24:57.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-01T05:24:57.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-04-01T05:24:57.787 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:24:57.787 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-04-01T05:24:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:24:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:24:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:24:58.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:24:58.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:24:58.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:24:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:24:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:24:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:24:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:24:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:24:59.895 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:25:00.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:25:00.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:25:00.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:25:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:25:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:25:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:25:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:25:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:25:00.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:25:00.418 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:25:00.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:25:00.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:25:00.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:25:00.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:25:00.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:25:00.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:25:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:00.442+0000 7f470260e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:00.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:00.444+0000 7f470260e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:00.446 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:00.445+0000 7f470260e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:00.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:25:00.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:25:00.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:25:00.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:25:00.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:25:00.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:00.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:25:00.709 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:25:00.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:00.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:00.960 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:00.959+0000 7f470260e640 -1 Falling back to public interface 2024-04-01T05:25:00.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:25:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:02.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:25:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:03.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:03.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:03.352+0000 7f470260e640 -1 osd.3 37 log_to_monitors true 2024-04-01T05:25:03.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:04.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:04.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:04.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:25:04.668 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:25:04.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:04.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:05.084 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/1303665019,v1:127.0.0.1:6827/1303665019] [v2:127.0.0.1:6828/1303665019,v1:127.0.0.1:6829/1303665019] exists,up 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:25:05.085 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:25:05.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:25:05.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:25:05.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:25:05.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:25:05.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:25:05.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:25:05.169 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:25:05.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:25:05.597 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:05.598 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:05.598 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:25:05.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:25:05.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:05.598 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:25:05.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-04-01T05:25:05.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-04-01T05:25:05.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-04-01T05:25:05.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:05.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:25:05.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-04-01T05:25:05.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-04-01T05:25:05.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672985' 2024-04-01T05:25:05.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:05.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:25:05.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-04-01T05:25:05.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-04-01T05:25:05.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672985 2-64424509460' 2024-04-01T05:25:05.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:05.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:25:06.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-04-01T05:25:06.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-04-01T05:25:06.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672985 2-64424509460 3-176093659138' 2024-04-01T05:25:06.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:06.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-04-01T05:25:06.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:06.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:25:06.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-04-01T05:25:06.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:06.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-04-01T05:25:06.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-01T05:25:06.030 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836509 2024-04-01T05:25:06.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:06.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-04-01T05:25:06.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:25:07.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:25:07.317 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:07.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-04-01T05:25:07.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:07.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-04-01T05:25:07.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:07.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:25:07.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-04-01T05:25:07.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:07.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-04-01T05:25:07.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-01T05:25:07.614 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672985 2024-04-01T05:25:07.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:25:07.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672985 2024-04-01T05:25:07.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:07.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-04-01T05:25:07.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:07.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:25:07.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-04-01T05:25:07.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:07.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-04-01T05:25:07.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-01T05:25:07.943 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509460 2024-04-01T05:25:07.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:25:08.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-04-01T05:25:08.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:08.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-176093659138 2024-04-01T05:25:08.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:08.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:25:08.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-176093659138 2024-04-01T05:25:08.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:08.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-04-01T05:25:08.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-04-01T05:25:08.229 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 176093659138 2024-04-01T05:25:08.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:25:08.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-04-01T05:25:08.513 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:25:08.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:08.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:08.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:25:08.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:25:08.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:25:08.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:25:08.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:25:08.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:25:08.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:25:08.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:25:09.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:25:09.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:25:09.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:09.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:25:09.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:25:09.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:25:09.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:25:09.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:25:09.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:25:09.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:09.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:09.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:09.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:10.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:10.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:25:10.390 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:25:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:10.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:10.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:24:50.479651+0000 '>' 2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:10.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:25:11.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:25:11.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:11.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:11.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:11.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:11.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:11.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:11.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:24:50.479651+0000 '>' 2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:11.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:25:12.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:25:12.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:12.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:12.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:12.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:12.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:12.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:13.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:24:50.479651+0000 '>' 2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:13.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:25:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:25:14.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:14.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:14.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:14.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:14.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:14.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:14.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:24:50.479651+0000 '>' 2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:14.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:25:15.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:25:15.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:15.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:15.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:15.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:15.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:15.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:10.485325+0000 '>' 2024-04-01T05:24:50.479651+0000 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:25:15.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:25:15.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:25:15.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:25:15.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:25:15.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:25:15.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:25:15.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:25:16.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:25:16.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:25:17.679 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:25:17.679 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:25:17.679 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:25:17.679 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:25:17.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:25:17.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:25:17.948 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:25:17.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:25:17.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:25:17.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:25:17.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:25:17.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:25:17.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:25:17.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:17.975+0000 7fd7ab3fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:17.977 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:17.978+0000 7fd7ab3fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:17.978 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:17.979+0000 7fd7ab3fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:18.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:25:18.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:25:18.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:25:18.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:25:18.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:25:18.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:18.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:25:18.286 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:25:18.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:18.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:18.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:18.495+0000 7fd7ab3fa640 -1 Falling back to public interface 2024-04-01T05:25:18.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:25:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:19.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:19.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:20.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:20.699+0000 7fd7ab3fa640 -1 osd.3 43 log_to_monitors true 2024-04-01T05:25:20.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:20.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:20.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:25:20.859 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:20.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:20.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:21.153 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 47 up_thru 41 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/1404402021,v1:127.0.0.1:6827/1404402021] [v2:127.0.0.1:6828/1404402021,v1:127.0.0.1:6829/1404402021] exists,up 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:25:21.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:25:21.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:25:21.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:25:21.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:25:21.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:25:21.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:25:21.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:25:21.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:25:21.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:25:21.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:21.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:25:21.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-04-01T05:25:21.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-04-01T05:25:21.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-04-01T05:25:21.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:21.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:25:21.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-04-01T05:25:21.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-04-01T05:25:21.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672990' 2024-04-01T05:25:21.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:21.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:25:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-04-01T05:25:22.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-04-01T05:25:22.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672990 2-64424509464' 2024-04-01T05:25:22.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:22.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:25:22.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462914 2024-04-01T05:25:22.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462914 2024-04-01T05:25:22.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672990 2-64424509464 3-201863462914' 2024-04-01T05:25:22.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:22.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-04-01T05:25:22.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:22.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:25:22.184 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-04-01T05:25:22.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:22.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-04-01T05:25:22.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-04-01T05:25:22.186 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836513 2024-04-01T05:25:22.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:22.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-04-01T05:25:22.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:25:23.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:25:23.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-04-01T05:25:23.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:23.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-04-01T05:25:23.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:23.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:25:23.765 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-04-01T05:25:23.766 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:23.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-04-01T05:25:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-04-01T05:25:23.767 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672990 2024-04-01T05:25:23.767 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:25:24.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672990 -lt 42949672990 2024-04-01T05:25:24.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:24.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-04-01T05:25:24.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:24.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:25:24.061 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-04-01T05:25:24.061 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:24.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-04-01T05:25:24.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-04-01T05:25:24.063 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509464 2024-04-01T05:25:24.063 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:25:24.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509464 2024-04-01T05:25:24.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:24.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462914 2024-04-01T05:25:24.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:24.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:25:24.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462914 2024-04-01T05:25:24.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:24.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462914 2024-04-01T05:25:24.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-04-01T05:25:24.350 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 201863462914 2024-04-01T05:25:24.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:25:24.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462914 -lt 201863462914 2024-04-01T05:25:24.631 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:25:24.631 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:24.631 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:25:24.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:25:25.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:25:25.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:25:25.341 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:25.341 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:25:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:25:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:25:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:25:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:25:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:25:25.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:25:25.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:25:25.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:25:25.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:25:25.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:25:25.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:25:25.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:25:25.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:25:26.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:25:26.075 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:25:26.075 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:26.075 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:26.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:25:26.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-01T05:25:26.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:25:26.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:25:26.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:25:26.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:25:26.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:25:26.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:25:26.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:25:26.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:25:26.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:25:26.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:25:26.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:25:26.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:25:26.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:25:26.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:25:27.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:25:27.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:25:27.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:27.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:27.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:25:27.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:25:27.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:25:27.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:25:27.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:25:27.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:25:27.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:25:27.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:25:27.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:27.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:25:27.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:25:27.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:25:27.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:25:27.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:25:27.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:25:29.777 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:25:30.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:25:30.297 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:25:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:25:30.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:25:30.323 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:30.323+0000 7efce0319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:30.326 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:30.326+0000 7efce0319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:30.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:30.327+0000 7efce0319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:30.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:30.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:30.842+0000 7efce0319640 -1 Falling back to public interface 2024-04-01T05:25:30.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:25:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:32.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:33.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:33.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:33.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:25:33.167 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:33.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:33.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:33.222 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:33.222+0000 7efce0319640 -1 osd.1 49 log_to_monitors true 2024-04-01T05:25:33.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:34.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:34.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:34.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:25:34.462 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:25:34.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:34.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/3929778057,v1:127.0.0.1:6811/3929778057] [v2:127.0.0.1:6812/3929778057,v1:127.0.0.1:6813/3929778057] exists,up 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:25:34.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:25:34.811 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:25:34.811 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:25:34.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:25:34.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:25:34.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:25:34.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:25:34.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:25:34.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:25:35.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:25:35.302 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:35.302 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:35.302 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:25:35.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:25:35.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:35.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:25:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836517 2024-04-01T05:25:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836517 2024-04-01T05:25:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517' 2024-04-01T05:25:35.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:35.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:25:35.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233986 2024-04-01T05:25:35.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233986 2024-04-01T05:25:35.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-231928233986' 2024-04-01T05:25:35.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:35.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:25:35.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-04-01T05:25:35.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-04-01T05:25:35.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-231928233986 2-64424509468' 2024-04-01T05:25:35.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:35.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:25:35.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462917 2024-04-01T05:25:35.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462917 2024-04-01T05:25:35.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-231928233986 2-64424509468 3-201863462917' 2024-04-01T05:25:35.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:35.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836517 2024-04-01T05:25:35.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:35.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:25:35.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836517 2024-04-01T05:25:35.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:35.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836517 2024-04-01T05:25:35.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-04-01T05:25:35.726 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836517 2024-04-01T05:25:35.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836517 2024-04-01T05:25:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:25:37.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:25:37.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:37.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836517 2024-04-01T05:25:37.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:37.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233986 2024-04-01T05:25:37.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:37.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:25:37.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233986 2024-04-01T05:25:37.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:37.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233986 2024-04-01T05:25:37.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-04-01T05:25:37.323 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 231928233986 2024-04-01T05:25:37.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:25:37.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233986 -lt 231928233986 2024-04-01T05:25:37.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:37.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-04-01T05:25:37.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:37.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:25:37.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-04-01T05:25:37.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:37.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-04-01T05:25:37.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-04-01T05:25:37.608 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509468 2024-04-01T05:25:37.608 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:25:37.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-04-01T05:25:37.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:37.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462917 2024-04-01T05:25:37.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:37.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:25:37.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462917 2024-04-01T05:25:37.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:37.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462917 2024-04-01T05:25:37.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462917' 2024-04-01T05:25:37.893 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 201863462917 2024-04-01T05:25:37.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:25:38.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462918 -lt 201863462917 2024-04-01T05:25:38.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:25:38.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:38.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:38.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:25:38.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:25:38.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:25:38.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:25:38.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:25:38.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:25:38.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:25:38.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:25:38.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:25:38.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:25:38.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:38.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:39.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:25:39.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:25:39.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:25:39.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:25:39.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:25:39.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:25:39.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:25:39.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:39.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:39.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:39.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:25:10.485325+0000 2024-04-01T05:25:39.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:25:40.111 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:25:40.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:25:10.485325+0000 2024-04-01T05:25:40.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:25:40.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:25:10.485325+0000 2024-04-01T05:25:40.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:25:40.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:25:40.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:40.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:40.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:40.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:40.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:40.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:10.485325+0000 '>' 2024-04-01T05:25:10.485325+0000 2024-04-01T05:25:40.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:25:41.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:25:41.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:25:41.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:25:41.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:25:41.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:25:41.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:25:41.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:25:41.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:40.824666+0000 '>' 2024-04-01T05:25:10.485325+0000 2024-04-01T05:25:41.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:25:41.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:25:41.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:25:41.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:25:41.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:25:42.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:25:43.425 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:25:43.425 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:25:43.426 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:25:43.426 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:25:43.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:43.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:25:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:25:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:25:43.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:25:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:25:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:25:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:25:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:25:43.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:43.722+0000 7f3b1be13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:43.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:43.725+0000 7f3b1be13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:43.726 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:43.726+0000 7f3b1be13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:43.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:25:43.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:44.242 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:44.240+0000 7f3b1be13640 -1 Falling back to public interface 2024-04-01T05:25:44.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:25:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:45.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:46.417 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:25:46.418+0000 7f3b1be13640 -1 osd.3 56 log_to_monitors true 2024-04-01T05:25:46.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:46.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:46.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:25:46.588 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:46.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:46.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 60 up_thru 54 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/2269112261,v1:127.0.0.1:6827/2269112261] [v2:127.0.0.1:6828/2269112261,v1:127.0.0.1:6829/2269112261] exists,up 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:25:46.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:25:46.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:25:46.921 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:25:46.921 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:25:46.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:25:46.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:25:46.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:25:47.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:25:47.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:25:47.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:25:47.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:25:47.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:25:47.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:3' 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:47.518 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:25:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-04-01T05:25:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-04-01T05:25:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-04-01T05:25:47.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:47.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:25:47.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233989 2024-04-01T05:25:47.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233989 2024-04-01T05:25:47.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-231928233989' 2024-04-01T05:25:47.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:47.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:25:47.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-04-01T05:25:47.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-04-01T05:25:47.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-231928233989 2-64424509472' 2024-04-01T05:25:47.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:47.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:25:48.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-04-01T05:25:48.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-04-01T05:25:48.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-231928233989 2-64424509472 3-257698037762' 2024-04-01T05:25:48.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:48.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-04-01T05:25:48.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:48.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:25:48.208 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-04-01T05:25:48.208 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:48.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-04-01T05:25:48.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-04-01T05:25:48.209 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836520 2024-04-01T05:25:48.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:48.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-04-01T05:25:48.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:25:49.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:25:49.518 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:25:49.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-04-01T05:25:49.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:49.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233989 2024-04-01T05:25:49.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:49.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:25:49.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233989 2024-04-01T05:25:49.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:49.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233989 2024-04-01T05:25:49.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233989' 2024-04-01T05:25:49.815 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 231928233989 2024-04-01T05:25:49.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:25:50.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233990 -lt 231928233989 2024-04-01T05:25:50.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:50.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-04-01T05:25:50.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:50.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:25:50.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-04-01T05:25:50.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:50.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-04-01T05:25:50.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-04-01T05:25:50.143 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509472 2024-04-01T05:25:50.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:25:50.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509472 -lt 64424509472 2024-04-01T05:25:50.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:25:50.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037762 2024-04-01T05:25:50.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:25:50.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:25:50.470 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037762 2024-04-01T05:25:50.470 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:25:50.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-04-01T05:25:50.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-04-01T05:25:50.471 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 257698037762 2024-04-01T05:25:50.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:25:50.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-04-01T05:25:50.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:25:50.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:50.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:51.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:25:51.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:25:51.129 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:25:51.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:25:51.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:25:51.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:25:51.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:25:51.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:25:51.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:25:51.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:25:51.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:25:51.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:25:51.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:25:51.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:25:51.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:25:51.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:25:51.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191705"' 2024-04-01T05:25:51.805 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191705' 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191707"' 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191707' 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191705 191707' 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191705 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191708: /' 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:25:51.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191710: /' 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:25:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:25:54.508 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:25:54.509 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:25:54.509 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:25:54.509 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:25:54.510 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: start osd.1 2024-04-01T05:25:54.511 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:25:54.665 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.665 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:25:54.665 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:25:54.665 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:25:54.665 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:25:54.666 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: remove 2#2:eb822e21:::SOMETHING:head# 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:25:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:25:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:25:54.669 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:25:54.670 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:25:54.670 INFO:tasks.workunit.client.0.smithi067.stderr:191710: start osd.2 2024-04-01T05:25:54.670 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:25:59.081 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:25:59.081 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:25:59.081 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:25:59.081 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:25:59.081 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:25:59.081 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2024-04-01T05:25:54.539+0000 7f0c7c316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2024-04-01T05:25:54.542+0000 7f0c7c316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2024-04-01T05:25:54.543+0000 7f0c7c316640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 0 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2024-04-01T05:25:55.078+0000 7f0c7c316640 -1 Falling back to public interface 2024-04-01T05:25:59.082 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 1 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.083 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2024-04-01T05:25:57.719+0000 7f0c7c316640 -1 osd.1 62 log_to_monitors true 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 3 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.084 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:25:59.298 INFO:tasks.workunit.client.0.smithi067.stderr:191708: osd.1 up in weight 1 up_from 67 up_thru 67 down_at 63 last_191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:25:59.298 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:25:59.298 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2024-04-01T05:25:54.694+0000 7fa58890f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2024-04-01T05:25:54.695+0000 7fa58890f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2024-04-01T05:25:54.696+0000 7fa58890f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 0 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.299 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2024-04-01T05:25:55.216+0000 7fa58890f640 -1 Falling back to public interface 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 1 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.300 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2024-04-01T05:25:58.080+0000 7fa58890f640 -1 osd.2 62 log_to_monitors true 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 3 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:25:59.301 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:25:59.785 INFO:tasks.workunit.client.0.smithi067.stderr:191710: osd.2 up in weight 1 up_from 69 up_thru 50 down_at 63 last_clean_interval [54,62) [v2:127.0.0.1:6810/3324070909,v1:127.0.0.1:6811/3324070909] [v2:127.0.0.1:6812/3324070909,v1:127.0.0.1:6813/3324070909] exists,up 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:25:59.785 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:25:59.785 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:25:59.785 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:25:59.785 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:25:59.785 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:25:59.786 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 1 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 2 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: 3' 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-04-01T05:25:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-287762808834' 2024-04-01T05:25:59.788 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,62) [v2:127.0.0.1:6818/574956672,v1:127.0.0.1:6819/574956672] [v2:127.0.0.1:6820/574956672,v1:127.0.0.1:6821/574956672] exists,up fa394218-fd37-4c9a-b739-88b5b9399a3d 2024-04-01T05:26:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:191710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:191710: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:00.112 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:00.113 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 1 2024-04-01T05:26:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 2 2024-04-01T05:26:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 3' 2024-04-01T05:26:00.114 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:00.115 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:00.115 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:00.115 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-04-01T05:26:00.115 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-04-01T05:26:00.115 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-04-01T05:26:00.115 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:00.116 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:00.116 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808835 2024-04-01T05:26:00.116 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808835 2024-04-01T05:26:00.117 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-287762808835' 2024-04-01T05:26:00.117 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph telltell osd.2 flush_pg_stats 2024-04-01T05:26:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-04-01T05:26:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-04-01T05:26:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-287762808834 2-296352743426' 2024-04-01T05:26:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037765 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037765 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-287762808834 2-296352743426 3-257698037765' 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: waiting osd.0 seq 21474836524 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836524 2024-04-01T05:26:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: waiting osd.1 seq 287762808834 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808834 2024-04-01T05:26:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:1917 osd.2 flush_pg_stats 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743427 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743427 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-287762808835 2-296352743427' 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-287762808835 2-296352743427 3-257698037766' 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: waiting osd.0 seq 21474836525 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836525 2024-04-01T05:26:02.502 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808835 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808835 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808835 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: waiting osd.1 seq 287762808835 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808835 -lt 287762808835 2024-04-01T05:26:02.503 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:03.414 INFO:tasks.workunit.client.0.smithi067.stderr:191710: 08: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743426 2024-04-01T05:26:03.414 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:03.414 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:03.414 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743426 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: waiting osd.2 seq 296352743426 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743426 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037765 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037765 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037765 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037765' 2024-04-01T05:26:03.415 INFO:tasks.workunit.client.0.smithi067.stderr:191708: waiting osd.3 seq 257698037765 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037765 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:03.416 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:03.417 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:03.774 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.c//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743427 2024-04-01T05:26:03.774 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:03.774 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743427 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743427 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: waiting osd.2 seq 296352743427 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743427 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037766 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037766 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-04-01T05:26:03.775 INFO:tasks.workunit.client.0.smithi067.stderr:191710: waiting osd.3 seq 257698037766 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037766 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:03.776 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:04.891 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.clienlient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:04.891 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:04.891 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:04.891 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:26:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/t.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:05.354 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:26:05.355 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:26:05.356 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clonclone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:05.766 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:191708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:26:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191707 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:e.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:06.189 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:191710: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:26:06.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:26:06.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:26:06.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:26:06.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:26:06.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:26:06.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:26:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:26:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:26:06.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:26:06.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:26:06.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:06.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:06.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:06.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:06.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:06.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:26:07.021 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:26:07.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:07.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:07.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:07.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:40.824666+0000 '>' 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:07.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:08.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:08.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:08.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:08.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:08.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:08.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:08.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:40.824666+0000 '>' 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:08.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:09.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:09.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:09.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:09.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:09.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:09.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:09.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:09.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:40.824666+0000 '>' 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:09.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:10.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:10.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:10.898 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:10.898 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:10.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:10.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:10.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:11.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:40.824666+0000 '>' 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:11.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:12.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:12.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:12.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:12.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:12.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:12.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:12.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:12.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:25:40.824666+0000 '>' 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:12.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:13.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:13.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:13.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:13.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:13.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:13.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:26:07.746372+0000 '>' 2024-04-01T05:25:40.824666+0000 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194826"' 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194826' 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194827"' 2024-04-01T05:26:13.769 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194827' 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 194826 194827' 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194826 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194829: /' 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:26:13.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194831: /' 2024-04-01T05:26:16.192 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.192 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:26:16.193 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: _ 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: hinfo_key 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: snapset 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:26:16.194 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:26:16.195 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:26:16.195 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:26:16.195 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:26:16.195 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:26:16.195 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:26:16.196 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:26:16.197 INFO:tasks.workunit.client.0.smithi067.stderr:194831: start osd.2 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:26:16.221 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:26:16.222 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: _ 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: hinfo_key 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: snapset 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:26:16.223 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:26:16.224 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:26:16.224 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:26:16.224 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:26:16.224 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:26:16.224 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:26:16.225 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:26:16.226 INFO:tasks.workunit.client.0.smithi067.stderr:194829: start osd.1 2024-04-01T05:26:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:26:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:26:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:26:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:26:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:26:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2024-04-01T05:26:16.225+0000 7f2f26810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2024-04-01T05:26:16.228+0000 7f2f26810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2024-04-01T05:26:16.230+0000 7f2f26810640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:26:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 0 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2024-04-01T05:26:16.742+0000 7f2f26810640 -1 Falling back to public interface 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:26:20.932 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 1 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:26:20.933 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2024-04-01T05:26:19.302+0000 7f2f26810640 -1 osd.2 70 log_to_monitors true 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 3 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194831: osd.2 up in weight 1 up_from 75 up_thru 75 down_aovery_ops 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2024-04-01T05:26:16.266+0000 7f1a98b45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2024-04-01T05:26:16.268+0000 7f1a98b45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:20.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2024-04-01T05:26:16.270+0000 7f1a98b45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 0 2024-04-01T05:26:20.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2024-04-01T05:26:16.788+0000 7f1a98b45640 -1 Falling back to public interface 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 1 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:20.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2024-04-01T05:26:19.525+0000 7f1a98b45640 -1 osd.1 70 log_to_monitors true 2024-04-01T05:26:20.938 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:20.939 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:20.939 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:26:20.939 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 3 2024-04-01T05:26:20.939 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:20.939 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:21.690 INFO:tasks.workunit.client.0.smithi067.stderr:194829: osd.1 up in weight 1 up_from 76 up_thru 69 down_at 71 last_clean_interval [69,70) [v2:127.0.0.1:6810/2495594032,v1:127.0.0.1:6811/2495594032] [v2:127.0.0.1:6812/2495594032,v1:127.0.0.1:6813/2495594032] exists,up fa394218-fd37-4c9a-b739-88b5b9399a3d 2024-04-01T05:26:21.690 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:21.691 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 1 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 2 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: 3' 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-04-01T05:26:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-326417514498' 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 71 last_clean_interval [67,70) [v2:127.0.0.1:6818/2336754594,v1:127.0.0.1:6819/2336754594] [v2:127.0.0.1:6820/2336754594,v1:127.0.0.1:6821/2336754594] exists,up 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:21.693 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 1 2024-04-01T05:26:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 2 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: 3' 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499' 2024-04-01T05:26:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-326417514498 2-322122547202' 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:23.929 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037773 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037773 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-326417514498 2-322122547202 3-257698037773' 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: waiting osd.0 seq 21474836531 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836531 2024-04-01T05:26:23.930 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514498 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514498 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: waiting osd.1 seq 326417514498 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:23.931 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-04-01T05:26:23.934 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-04-01T05:26:23.934 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-04-01T05:26:23.934 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-04-01T05:26:23.934 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499 2-322122547203' 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499 2-322122547203 3-257698037772' 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-04-01T05:26:23.935 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: waiting osd.0 seq 21474836530 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836530 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514499 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514499 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: waiting osd.1 seq 326417514499 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547202 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547202 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-04-01T05:26:25.183 INFO:tasks.workunit.client.0.smithi067.stderr:194831: waiting osd.2 seq 322122547202 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037773 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037773 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037773 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: waiting osd.3 seq 257698037773 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:25.184 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037773 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:25.185 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephte$seqs 2024-04-01T05:26:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547203 2024-04-01T05:26:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:25.238 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547203 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547203' 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: waiting osd.2 seq 322122547203 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037772 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:25.239 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037772 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: waiting osd.3 seq 257698037772 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037772 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:25.240 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:25.241 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:25.241 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:25.241 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:25.241 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:25.241 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:25.241 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:194831: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:194831: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:25.620 INFO:tasks.workunit.client.0.smithi067.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:25.620 INFO:tasks.workunit.client.0.smithi067.stderr:194829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:194829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194827 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:26:25.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:26:25.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:26:25.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:26:25.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:26:25.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:26:25.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:26:25.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197505"' 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197505' 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197507"' 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197507' 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 197505 197507' 2024-04-01T05:26:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:26:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:26:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197505 2024-04-01T05:26:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197508: /' 2024-04-01T05:26:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:26:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197510: /' 2024-04-01T05:26:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.414 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:26:28.415 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:26:28.416 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:26:28.417 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:26:28.417 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:26:28.417 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:26:28.417 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:26:28.418 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: start osd.1 2024-04-01T05:26:28.419 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:26:28.945 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.945 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:26:28.945 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:26:28.946 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:26:28.947 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:26:28.948 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:26:28.949 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: start osd.3 2024-04-01T05:26:28.950 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:26:32.962 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2024-04-01T05:26:28.446+0000 7f0fc740e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2024-04-01T05:26:28.448+0000 7f0fc740e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2024-04-01T05:26:28.450+0000 7f0fc740e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:26:32.963 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 0 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2024-04-01T05:26:28.969+0000 7f0fc740e640 -1 Falling back to public interface 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 1 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:32.964 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2024-04-01T05:26:31.584+0000 7f0fc740e640 -1 osd.1 78 log_to_monitors true 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 3 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:32.965 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:33.452 INFO:tasks.workunit.client.0.smithi067.stderr:197510: osd.1 up in weight 1 up_from 83 up_thru 83 down_at 79 last_197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:26:33.452 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2024-04-01T05:26:28.973+0000 7f2030cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2024-04-01T05:26:28.975+0000 7f2030cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2024-04-01T05:26:28.976+0000 7f2030cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:26:33.453 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 0 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2024-04-01T05:26:29.492+0000 7f2030cb5640 -1 Falling back to public interface 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 1 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:33.454 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2024-04-01T05:26:32.145+0000 7f2030cb5640 -1 osd.3 78 log_to_monitors true 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 3 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:33.455 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:33.860 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [76,78) [v2:127.0.0.1:6818/262446731,v1:127.0.0.1:6819/262446731] [v2:127.0.0.1:6820/262446731,v1:127.0.0.1:6821/262446731] exists,up 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:26:33.860 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:33.860 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:33.860 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:33.860 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:33.861 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 1 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 2 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: 3' 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-04-01T05:26:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570' 2024-04-01T05:26:33.863 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:35.275 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tells.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:35.275 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:35.275 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 4 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: osd.3 up in weight 1 up_from 86 up_thru 86 down_at 79 last_clean_interval [60,78) [v2:127.0.0.1:6826/3255552027,v1:127.0.0.1:6827/3255552027] [v2:127.0.0.1:6828/3255552027,v1:127.0.0.1:6829/3255552027] exists,up 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:35.276 INFO:tasks.workunit.client.0.smithi067.stderr:197508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 1 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 2 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: 3' 2024-04-01T05:26:35.277 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:35.278 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:35.278 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:35.278 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-04-01T05:26:35.278 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-04-01T05:26:35.278 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-04-01T05:26:35.278 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel osd.2 flush_pg_stats 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547206 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547206 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570 2-322122547206' 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187458 2024-04-01T05:26:36.078 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187458 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570 2-322122547206 3-369367187458' 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: waiting osd.0 seq 21474836534 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836534 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: waiting osd.1 seq 356482285570 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-04-01T05:26:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:37.240 INFO:tasks.workunit.client.0.smithi067.stderr:197510: pers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:37.240 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-04-01T05:26:37.240 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-356482285571' 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547208 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547208 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-356482285571 2-322122547208' 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187459 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187459 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-356482285571 2-322122547208 3-369367187459' 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-04-01T05:26:37.241 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: waiting osd.0 seq 21474836536 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836536 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285571 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:37.242 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:37.243 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285571 2024-04-01T05:26:37.243 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/u//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547206 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547206 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547206 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547206' 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: waiting osd.2 seq 322122547206 2024-04-01T05:26:37.313 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547207 -lt 322122547206 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187458 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187458 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187458 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187458' 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: waiting osd.3 seq 369367187458 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187458 -lt 369367187458 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:37.314 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:37.315 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:37.699 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:37.699 INFO:tasks.workunit.client.0.smithi067.stderr:197510: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:37.699 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:37.700 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:37.700 INFO:tasks.workunit.client.0.smithi067.stderr:197510: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:37.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: waiting osd.1 seq 356482285571 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285572 -lt 356482285571 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547208 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547208 2024-04-01T05:26:38.528 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547208 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547208' 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: waiting osd.2 seq 322122547208 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547208 -lt 322122547208 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187459 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187459 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187459 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-04-01T05:26:38.529 INFO:tasks.workunit.client.0.smithi067.stderr:197508: waiting osd.3 seq 369367187459 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187459 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:38.530 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:39.186 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:197508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197507 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:26:39.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:26:39.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:26:39.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:26:39.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:26:39.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:26:39.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:26:39.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:26:39.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:26:39.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:26:39.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:39.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:39.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:39.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:39.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:39.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:26:40.017 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:40.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:40.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:26:07.746372+0000 '>' 2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:40.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:41.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:41.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:26:07.746372+0000 '>' 2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:41.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:42.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:42.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:42.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:42.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:42.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:42.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:42.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:42.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:26:07.746372+0000 '>' 2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:42.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:43.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:43.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:43.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:43.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:43.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:43.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:43.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:26:07.746372+0000 '>' 2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:26:45.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:26:45.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:26:45.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:26:45.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:26:45.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:26:45.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:26:45.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:26:45.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:26:40.274275+0000 '>' 2024-04-01T05:26:07.746372+0000 2024-04-01T05:26:45.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:26:45.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:26:45.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:26:45.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:26:45.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:26:45.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200381"' 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200381' 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200383"' 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200383' 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:26:45.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 200381 200383' 2024-04-01T05:26:45.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:26:45.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:26:45.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200381 2024-04-01T05:26:45.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200384: /' 2024-04-01T05:26:45.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:45.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200386: /' 2024-04-01T05:26:47.991 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:26:47.991 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:26:47.991 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:26:47.991 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:26:47.992 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: _ 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: hinfo_key 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: snapset 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:26:47.993 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:26:47.994 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:26:47.995 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:26:47.996 INFO:tasks.workunit.client.0.smithi067.stderr:200384: start osd.3 2024-04-01T05:26:48.267 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:26:48.268 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: _ 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: hinfo_key 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: snapset 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:26:48.269 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:26:48.270 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:26:48.271 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:26:48.272 INFO:tasks.workunit.client.0.smithi067.stderr:200386: start osd.1 2024-04-01T05:26:52.171 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:26:52.171 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:26:52.171 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:26:52.171 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2024-04-01T05:26:48.024+0000 7f124cc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2024-04-01T05:26:48.030+0000 7f124cc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2024-04-01T05:26:48.031+0000 7f124cc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 0 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:52.172 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2024-04-01T05:26:48.545+0000 7f124cc0c640 -1 Falling back to public interface 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 1 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2024-04-01T05:26:51.022+0000 7f124cc0c640 -1 osd.3 88 log_to_monitors true 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:52.173 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:52.174 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:52.174 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:26:52.174 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 3 2024-04-01T05:26:52.174 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2024-04-01T05:26:48.296+0000 7fa6d100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2024-04-01T05:26:48.299+0000 7fa6d100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2024-04-01T05:26:48.300+0000 7fa6d100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:26:53.003 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 0 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2024-04-01T05:26:48.816+0000 7fa6d100d640 -1 Falling back to public interface 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 1 2024-04-01T05:26:53.004 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2024-04-01T05:26:51.530+0000 7fa6d100d640 -1 osd.1 88 log_to_monitors true 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 3 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:26:53.005 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200386: osd.1 up in weight 1 up_from 96 up_thru 83 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: osd.3 up in weight 1 up_from 93 up_thru 93 down_at 89 last_clean_interval [86,88) [v2:127.0.0.1:6818/734674647,v1:127.0.0.1:6819/734674647] [v2:127.0.0.1:6820/734674647,v1:127.0.0.1:6821/734674647] exists,up 53d5d26f-3746-48be-b55b-aae15a84b6cf 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:53.492 INFO:tasks.workunit.client.0.smithi067.stderr:200384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 1 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 2 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: 3' 2024-04-01T05:26:53.493 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-04-01T05:26:53.494 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-412316860418' 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $it 89 last_clean_interval [83,88) [v2:127.0.0.1:6826/145919710,v1:127.0.0.1:6827/145919710] [v2:127.0.0.1:6828/145919710,v1:127.0.0.1:6829/145919710] exists,up 0b7d1c55-e678-477c-873f-c18edafa5932 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:26:53.837 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 1 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 2 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: 3' 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:26:53.838 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-412316860419' 2024-04-01T05:26:53.839 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ds 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547212 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547212 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-412316860418 2-322122547212' 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-412316860418 2-322122547212 3-399431958530' 2024-04-01T05:26:55.874 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: waiting osd.0 seq 21474836540 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836540 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:55.875 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836540 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-412316860418 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-412316860418 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 412316860418' 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: waiting osd.1 seq 412316860418 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:55.876 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-04-01T05:26:56.074 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /h ceph tell osd.2 flush_pg_stats 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547213 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547213 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-412316860419 2-322122547213' 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-412316860419 2-322122547213 3-399431958531' 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:26:56.075 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: waiting osd.0 seq 21474836541 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836541 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836541 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-412316860419 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:26:56.076 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-412316860419 2024-04-01T05:26:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-04-01T05:26:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 412316860419' 2024-04-01T05:26:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:200386: waiting osd.1 seq 412316860419 2024-04-01T05:26:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:26:56.077 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547212 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547212 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547212 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547212' 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: waiting osd.2 seq 322122547212 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547214 -lt 322122547212 2024-04-01T05:26:57.119 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958530 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958530 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958530' 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: waiting osd.3 seq 399431958530 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:57.120 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:57.121 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.cliens 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547213 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547213 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547213 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547213' 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: waiting osd.2 seq 322122547213 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547214 -lt 322122547213 2024-04-01T05:26:57.340 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958531 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958531 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958531' 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: waiting osd.3 seq 399431958531 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/ct.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:58.618 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:26:58.619 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:26:58.620 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:26:58.620 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:26:58.620 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:26:58.620 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:26:58.620 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-04-01T05:26:58.801 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/celone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:58.802 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:58.803 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-04-01T05:26:58.804 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-04-01T05:26:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-01T05:26:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:59.542 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:26:59.543 INFO:tasks.workunit.client.0.smithi067.stderr:200386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:26:59.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:ph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:26:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:26:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:26:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:27:00.458 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:27:00.458 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:27:00.458 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:27:00.458 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:27:00.458 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:27:00.458 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:200384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200383 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:27:00.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:27:00.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:27:00.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:27:00.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:27:00.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:27:00.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:27:00.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:27:00.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:27:00.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:27:00.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:27:00.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:27:00.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:27:00.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:27:00.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:27:00.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:27:00.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:27:00.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:27:00.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:27:00.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:27:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:27:00.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:27:00.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:27:00.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:00.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:00.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:27:00.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:27:00.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:27:00.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:27:00.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:27:00.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:27:00.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:27:00.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:27:00.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:27:00.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:27:00.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:27:00.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:27:00.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:27:00.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:27:00.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:27:00.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:27:00.656 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:27:00.656 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:00.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:00.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:27:00.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:27:00.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:27:00.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:27:00.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:27:00.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:00.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:00.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-04-01T05:27:00.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:27:00.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:27:01.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:27:01.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:27:01.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:27:01.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:27:01.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:27:01.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:27:01.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:27:01.286 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:27:01.287 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:01.287 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:01.287 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:27:01.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:27:01.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:27:01.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:27:02.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:27:02.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:27:02.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:27:02.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-04-01T05:27:02.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:27:02.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-04-01T05:27:02.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:27:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:27:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:27:02.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:27:02.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:27:02.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:27:02.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:27:02.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:27:02.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7f42c13-571c-4ffe-8bd0-fa23b001388d 2024-04-01T05:27:02.221 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 a7f42c13-571c-4ffe-8bd0-fa23b001388d 2024-04-01T05:27:02.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a7f42c13-571c-4ffe-8bd0-fa23b001388d' 2024-04-01T05:27:02.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:27:02.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmRQpmNLcXDhAAm11YNpBH+zd51jF0Pv4KQQ== 2024-04-01T05:27:02.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmRQpmNLcXDhAAm11YNpBH+zd51jF0Pv4KQQ=="}' 2024-04-01T05:27:02.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7f42c13-571c-4ffe-8bd0-fa23b001388d -i td/osd-scrub-repair/0/new.json 2024-04-01T05:27:02.441 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:27:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:27:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmRQpmNLcXDhAAm11YNpBH+zd51jF0Pv4KQQ== --osd-uuid a7f42c13-571c-4ffe-8bd0-fa23b001388d 2024-04-01T05:27:02.481 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.481+0000 7fa411a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.492+0000 7fa411a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.494+0000 7fa411a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.494+0000 7fa411a11640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:27:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.494+0000 7fa411a11640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:27:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.494+0000 7fa411a11640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:27:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:02.584+0000 7fa411a11640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:27:08.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:27:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:27:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:27:08.195 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:27:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:27:08.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:27:08.566 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:27:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:27:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:27:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:27:08.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:27:08.592 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:08.591+0000 7f8ed1211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:08.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:08.594+0000 7f8ed1211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:08.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:08.595+0000 7f8ed1211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:08.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:27:08.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:27:08.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:27:08.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:27:08.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:27:08.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:08.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:27:08.854 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:27:08.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:08.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:09.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:09.109+0000 7f8ed1211640 -1 Falling back to public interface 2024-04-01T05:27:09.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:10.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:10.135 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:27:10.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:10.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:27:10.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:10.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:10.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:11.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:11.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:11.429 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:27:11.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:27:11.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:11.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:11.440 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:11.439+0000 7f8ed1211640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:27:11.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:12.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:13.699 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:27:13.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:13.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:27:13.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:13.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:13.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:14.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:14.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:14.161 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:27:14.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:27:14.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:14.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:14.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:15.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:15.455 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:27:15.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:15.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:27:15.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:15.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:15.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:16.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:16.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:16.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:27:16.752 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:27:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:16.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:17.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:18.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:18.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:18.128 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:27:18.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:27:18.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:18.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:18.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:19.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:19.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:19.423 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:27:19.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:27:19.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:19.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:19.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:20.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:20.717 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:27:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:27:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:20.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:21.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:22.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:22.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:22.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:27:22.015 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:27:22.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:22.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:22.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:23.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:23.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:23.804 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:27:23.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:27:23.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:23.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:24.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:25.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:27:25.391 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:27:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:25.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:25.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:26.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:27:26.733 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:27:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:26.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:27.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:28.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:28.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:28.031 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:27:28.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:27:28.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:28.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:28.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:29.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:29.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:29.443 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:27:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:27:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:29.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:30.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:31.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:31.138 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:27:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:27:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:31.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:31.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:32.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:27:32.691 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:27:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:32.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:27:33.020 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/787862928,v1:127.0.0.1:6803/787862928] [v2:127.0.0.1:6804/787862928,v1:127.0.0.1:6805/787862928] exists,up a7f42c13-571c-4ffe-8bd0-fa23b001388d 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:27:33.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:27:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:27:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e69106f6-d816-41af-b580-68818bbfdd8e 2024-04-01T05:27:33.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e69106f6-d816-41af-b580-68818bbfdd8e' 2024-04-01T05:27:33.028 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 e69106f6-d816-41af-b580-68818bbfdd8e 2024-04-01T05:27:33.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:27:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFRQpmG1yMAhAAsQw042twHu3C6F6oQRrCyQ== 2024-04-01T05:27:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFRQpmG1yMAhAAsQw042twHu3C6F6oQRrCyQ=="}' 2024-04-01T05:27:33.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e69106f6-d816-41af-b580-68818bbfdd8e -i td/osd-scrub-repair/1/new.json 2024-04-01T05:27:33.485 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:27:33.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:27:33.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFRQpmG1yMAhAAsQw042twHu3C6F6oQRrCyQ== --osd-uuid e69106f6-d816-41af-b580-68818bbfdd8e 2024-04-01T05:27:33.524 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.525+0000 7f8a0e9bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:33.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.569+0000 7f8a0e9bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.570+0000 7f8a0e9bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.571+0000 7f8a0e9bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:27:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.571+0000 7f8a0e9bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:27:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.571+0000 7f8a0e9bd640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:27:33.645 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:33.645+0000 7f8a0e9bd640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:27:39.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:27:39.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:27:39.711 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:27:39.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:27:39.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:27:40.078 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:27:40.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:27:40.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:27:40.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:27:40.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:27:40.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:27:40.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:40.104+0000 7f414ea0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:40.106 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:40.107+0000 7f414ea0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:40.107 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:40.108+0000 7f414ea0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:27:40.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:27:40.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:27:40.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:27:40.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:27:40.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:27:40.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:40.365 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:27:40.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:27:40.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:40.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:40.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:40.622+0000 7f414ea0d640 -1 Falling back to public interface 2024-04-01T05:27:40.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:41.648 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:27:41.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:27:41.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:41.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:41.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:42.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:42.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:42.943 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:27:42.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:27:42.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:42.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:42.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:27:42.967+0000 7f414ea0d640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:27:43.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:44.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:44.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:44.275 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:27:44.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:27:44.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:44.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:45.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:45.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:45.611 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:27:45.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:27:45.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:45.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:45.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:46.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:46.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:46.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:27:46.905 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:27:46.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:46.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:47.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:48.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:48.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:48.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:27:48.199 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:27:48.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:48.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:48.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:49.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:49.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:49.494 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:27:49.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:27:49.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:49.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:49.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:50.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:50.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:50.855 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:27:50.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:27:50.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:50.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:51.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:52.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:52.151 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:27:52.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:52.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:27:52.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:52.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:52.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:53.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:53.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:53.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:27:53.446 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:27:53.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:53.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:54.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:54.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:54.862 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:27:54.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:27:54.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:54.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:55.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:56.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:56.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:56.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:27:56.449 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:27:56.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:56.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:56.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:57.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:57.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:57.856 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:27:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:27:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:57.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:27:58.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:27:59.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:27:59.469 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:27:59.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:27:59.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:27:59.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:27:59.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:28:00.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:01.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:01.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:01.010 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:28:01.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:28:01.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:01.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:28:01.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:02.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:02.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:02.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:28:02.693 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:28:02.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:02.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:28:03.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:04.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:04.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:04.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:28:04.278 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:28:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:28:04.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:05.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:05.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:05.607 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:28:05.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:28:05.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:05.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:28:05.901 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/418128961,v1:127.0.0.1:6811/418128961] [v2:127.0.0.1:6812/418128961,v1:127.0.0.1:6813/418128961] exists,up e69106f6-d816-41af-b580-68818bbfdd8e 2024-04-01T05:28:05.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:28:05.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:28:05.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:28:05.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:28:05.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-04-01T05:28:05.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:28:05.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:28:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:28:05.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:28:05.908 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:28:05.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcee56f2-4788-485b-908d-b9e7ad90588a 2024-04-01T05:28:05.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fcee56f2-4788-485b-908d-b9e7ad90588a' 2024-04-01T05:28:05.908 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 fcee56f2-4788-485b-908d-b9e7ad90588a 2024-04-01T05:28:05.908 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:28:05.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlRQpm4OASNxAAJm2QEhBEiHMkt3HeSY3NUg== 2024-04-01T05:28:05.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlRQpm4OASNxAAJm2QEhBEiHMkt3HeSY3NUg=="}' 2024-04-01T05:28:05.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcee56f2-4788-485b-908d-b9e7ad90588a -i td/osd-scrub-repair/2/new.json 2024-04-01T05:28:06.347 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:28:06.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:28:06.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlRQpm4OASNxAAJm2QEhBEiHMkt3HeSY3NUg== --osd-uuid fcee56f2-4788-485b-908d-b9e7ad90588a 2024-04-01T05:28:06.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.387+0000 7fc27321b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:06.432 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.432+0000 7fc27321b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.433+0000 7fc27321b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.434+0000 7fc27321b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:28:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.434+0000 7fc27321b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:28:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.434+0000 7fc27321b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:28:06.504 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:06.504+0000 7fc27321b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:28:12.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:28:12.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:28:12.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:28:12.584 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:28:12.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:28:12.958 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:28:12.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:28:12.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:28:12.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:28:12.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:28:12.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:28:12.984 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:12.985+0000 7fb7e491f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:12.987+0000 7fb7e491f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:12.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:12.988+0000 7fb7e491f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:13.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:28:13.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:28:13.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:28:13.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:28:13.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:28:13.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:13.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:28:13.247 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:28:13.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:13.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:13.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:13.502+0000 7fb7e491f640 -1 Falling back to public interface 2024-04-01T05:28:13.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:14.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:14.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:14.535 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:28:14.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:28:14.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:14.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:14.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:15.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:15.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:15.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:28:15.824 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:28:15.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:15.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:15.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:15.847+0000 7fb7e491f640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:28:16.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:17.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:17.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:17.147 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:28:17.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:28:17.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:17.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:17.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:18.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:18.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:18.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:28:18.555 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:28:18.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:18.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:18.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:19.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:19.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:19.878 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:28:19.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:28:19.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:19.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:20.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:21.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:21.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:21.171 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:28:21.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:28:21.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:21.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:21.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:22.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:22.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:22.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:28:22.547 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:28:22.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:22.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:22.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:23.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:23.920 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:28:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:28:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:24.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:25.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:25.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:25.217 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:28:25.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:28:25.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:25.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:25.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:26.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:26.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:26.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:28:26.546 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:28:26.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:26.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:26.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:27.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:27.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:27.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:28:27.900 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:28:27.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:27.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:28.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:29.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:29.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:29.287 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:28:29.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:28:29.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:29.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:29.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:30.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:30.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:30.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:28:30.917 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:28:30.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:30.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:31.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:32.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:32.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:32.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:28:32.653 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:28:32.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:32.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:33.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:34.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:34.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:34.299 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:28:34.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:28:34.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:34.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:34.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:35.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:35.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:35.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:28:35.918 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:28:35.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:35.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:36.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:37.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:37.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:37.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:28:37.655 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:28:37.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:37.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:38.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:39.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:39.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:39.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:28:39.135 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:28:39.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:39.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:28:39.429 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/961833254,v1:127.0.0.1:6819/961833254] [v2:127.0.0.1:6820/961833254,v1:127.0.0.1:6821/961833254] exists,up fcee56f2-4788-485b-908d-b9e7ad90588a 2024-04-01T05:28:39.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:28:39.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:28:39.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:28:39.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:28:39.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:28:39.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:28:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:28:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d27cca05-2607-451e-8a01-4ff3f81a18c8' 2024-04-01T05:28:39.437 INFO:tasks.workunit.client.0.smithi067.stdout:add osd3 d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:28:39.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:28:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHRgpmQm3dGhAASc+Q6Wl7NczP1sRbOevKYQ== 2024-04-01T05:28:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHRgpmQm3dGhAASc+Q6Wl7NczP1sRbOevKYQ=="}' 2024-04-01T05:28:39.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d27cca05-2607-451e-8a01-4ff3f81a18c8 -i td/osd-scrub-repair/3/new.json 2024-04-01T05:28:39.785 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:28:39.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-04-01T05:28:39.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHRgpmQm3dGhAASc+Q6Wl7NczP1sRbOevKYQ== --osd-uuid d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:28:39.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.824+0000 7fb1bfc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:39.869 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.869+0000 7fb1bfc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:39.870 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.870+0000 7fb1bfc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:39.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.870+0000 7fb1bfc15640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:28:39.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.870+0000 7fb1bfc15640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:28:39.871 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.870+0000 7fb1bfc15640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:28:39.944 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:39.944+0000 7fb1bfc15640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-04-01T05:28:45.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-04-01T05:28:45.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:28:45.993 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd3 key to auth repository 2024-04-01T05:28:45.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-01T05:28:45.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:28:46.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-01T05:28:46.363 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.3 2024-04-01T05:28:46.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:28:46.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:28:46.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:28:46.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:28:46.388 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:46.389+0000 7f1474819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:46.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:46.402+0000 7f1474819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:46.402 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:46.403+0000 7f1474819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:28:46.655 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:28:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:46.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:46.931 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:46.931+0000 7f1474819640 -1 Falling back to public interface 2024-04-01T05:28:46.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:47.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:47.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:47.938 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:28:47.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:28:47.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:47.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:48.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:49.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:49.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:49.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:28:49.229 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:28:49.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:49.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:49.310 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:28:49.311+0000 7f1474819640 -1 osd.3 0 log_to_monitors true 2024-04-01T05:28:49.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:50.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:50.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:28:50.591 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:28:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:50.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:51.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:28:51.880 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:28:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:51.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:52.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:53.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:53.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:53.285 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:28:53.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:28:53.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:53.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:53.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:54.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:54.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:54.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:28:54.626 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:28:54.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:54.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:54.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:55.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:55.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:55.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:28:55.917 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:28:55.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:55.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:56.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:28:57.307 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:28:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:57.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:58.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:58.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:58.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:28:58.606 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:28:58.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:58.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:28:58.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:28:59.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:28:59.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:28:59.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:28:59.897 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:28:59.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:28:59.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:00.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:01.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:01.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:01.295 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:29:01.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:29:01.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:01.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:01.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:02.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:02.940 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:29:02.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:29:02.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:02.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:03.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:04.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:04.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:04.676 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:29:04.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:29:04.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:04.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:05.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:06.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:06.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:06.598 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:29:06.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:29:06.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:06.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:07.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:08.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:08.451 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:29:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:29:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:08.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:08.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:09.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:09.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:09.845 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:29:09.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:29:09.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:09.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:10.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:11.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:11.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:11.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:29:11.591 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:29:11.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:11.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:12.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:13.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:13.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:13.088 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:29:13.089 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:29:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:13.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:29:13.380 INFO:tasks.workunit.client.0.smithi067.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4137194765,v1:127.0.0.1:6827/4137194765] [v2:127.0.0.1:6828/4137194765,v1:127.0.0.1:6829/4137194765] exists,up d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:29:13.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:29:13.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:29:13.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab20b411-a139-4213-8855-8c720473596a 2024-04-01T05:29:13.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 ab20b411-a139-4213-8855-8c720473596a' 2024-04-01T05:29:13.388 INFO:tasks.workunit.client.0.smithi067.stdout:add osd4 ab20b411-a139-4213-8855-8c720473596a 2024-04-01T05:29:13.388 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:29:13.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApRgpmrl31FxAABlX1r7WSZ75UEdARqcALEg== 2024-04-01T05:29:13.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApRgpmrl31FxAABlX1r7WSZ75UEdARqcALEg=="}' 2024-04-01T05:29:13.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab20b411-a139-4213-8855-8c720473596a -i td/osd-scrub-repair/4/new.json 2024-04-01T05:29:13.763 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:29:13.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-04-01T05:29:13.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApRgpmrl31FxAABlX1r7WSZ75UEdARqcALEg== --osd-uuid ab20b411-a139-4213-8855-8c720473596a 2024-04-01T05:29:13.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.803+0000 7fa88ea1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:13.847 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.848+0000 7fa88ea1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:13.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.849+0000 7fa88ea1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:13.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.849+0000 7fa88ea1a640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-04-01T05:29:13.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.849+0000 7fa88ea1a640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-04-01T05:29:13.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.849+0000 7fa88ea1a640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-04-01T05:29:13.922 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:13.923+0000 7fa88ea1a640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-04-01T05:29:19.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-04-01T05:29:19.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:29:19.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-01T05:29:19.865 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd4 key to auth repository 2024-04-01T05:29:19.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:29:20.236 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.4 2024-04-01T05:29:20.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-01T05:29:20.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:29:20.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:29:20.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:29:20.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:29:20.264 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:20.263+0000 7ff9ab3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:20.265 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:20.265+0000 7ff9ab3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:20.266 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:20.266+0000 7ff9ab3ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-01T05:29:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:29:20.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:20.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:20.788 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:20.788+0000 7ff9ab3ac640 -1 Falling back to public interface 2024-04-01T05:29:20.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:21.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:21.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:21.840 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:29:21.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:29:21.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:21.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:22.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:23.133 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:23.132+0000 7ff9ab3ac640 -1 osd.4 0 log_to_monitors true 2024-04-01T05:29:23.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:23.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:23.134 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:29:23.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:29:23.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:23.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:23.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:24.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:24.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:24.467 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:29:24.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:29:24.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:24.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:24.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:25.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:25.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:25.765 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:29:25.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:29:25.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:25.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:27.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:27.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:27.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:29:27.156 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:29:27.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:27.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:27.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:28.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:28.545 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:29:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:29:28.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:28.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:28.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:29.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:29.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:29.901 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:29:29.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:29:29.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:29.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:30.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:31.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:31.192 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:29:31.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:31.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:29:31.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:31.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:31.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:32.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:32.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:32.540 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:29:32.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:29:32.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:32.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:32.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:33.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:33.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:33.954 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:29:33.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:29:33.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:33.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:34.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:35.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:35.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:35.248 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:29:35.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:29:35.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:35.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:36.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:37.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:37.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:37.050 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:29:37.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:29:37.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:37.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:37.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:38.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:38.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:38.852 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:29:38.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:29:38.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:38.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:39.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:40.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:40.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:40.145 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:29:40.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:29:40.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:40.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:40.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:41.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:41.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:41.564 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:29:41.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:29:41.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:41.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:42.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:43.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:43.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:43.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:29:43.128 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:29:43.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:43.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:43.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:44.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:44.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:44.739 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:29:44.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:29:44.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:44.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:45.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:46.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:29:46.056 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:29:46.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:46.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:29:46.362 INFO:tasks.workunit.client.0.smithi067.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1834401273,v1:127.0.0.1:6835/1834401273] [v2:127.0.0.1:6836/1834401273,v1:127.0.0.1:6837/1834401273] exists,up ab20b411-a139-4213-8855-8c720473596a 2024-04-01T05:29:46.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:29:46.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:29:46.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:29:46.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:29:46.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-04-01T05:29:46.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:29:46.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:29:46.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 755f7163-e86b-437f-a28a-2f0e44cec5c4' 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stdout:add osd5 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:29:46.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:29:46.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKRgpm7HreFhAAuqwvqXYugbLcC8qrf5AdHA== 2024-04-01T05:29:46.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKRgpm7HreFhAAuqwvqXYugbLcC8qrf5AdHA=="}' 2024-04-01T05:29:46.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 755f7163-e86b-437f-a28a-2f0e44cec5c4 -i td/osd-scrub-repair/5/new.json 2024-04-01T05:29:46.742 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:29:46.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-04-01T05:29:46.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKRgpm7HreFhAAuqwvqXYugbLcC8qrf5AdHA== --osd-uuid 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:29:46.782 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.782+0000 7f337100c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:46.827 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.827+0000 7f337100c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:46.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.828+0000 7f337100c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:46.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.828+0000 7f337100c640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-04-01T05:29:46.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.828+0000 7f337100c640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-04-01T05:29:46.828 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.828+0000 7f337100c640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-04-01T05:29:46.902 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:46.902+0000 7f337100c640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-04-01T05:29:53.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-04-01T05:29:53.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:29:53.140 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd5 key to auth repository 2024-04-01T05:29:53.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-01T05:29:53.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:29:53.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-01T05:29:53.513 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.5 2024-04-01T05:29:53.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:29:53.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:29:53.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:29:53.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:29:53.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:53.540+0000 7f74322ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:53.541 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:53.542+0000 7f74322ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:53.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:53.543+0000 7f74322ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:53.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:29:53.798 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:29:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:29:54.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:54.056+0000 7f74322ac640 -1 Falling back to public interface 2024-04-01T05:29:54.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:55.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:29:55.078 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:29:55.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:55.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:29:55.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:56.364 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:29:56.364+0000 7f74322ac640 -1 osd.5 0 log_to_monitors true 2024-04-01T05:29:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:56.402 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:29:56.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:29:56.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:56.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:29:56.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:57.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:57.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:57.714 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:29:57.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:29:57.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:57.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:29:58.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:29:59.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:29:59.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:29:59.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:29:59.007 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:29:59.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:29:59.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:29:59.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:00.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:00.377 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:30:00.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:00.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:30:00.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:00.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:00.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:01.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:01.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:01.672 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:30:01.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:30:01.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:01.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:01.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:02.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:02.966 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:30:02.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:30:02.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:02.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:03.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:04.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:30:04.278 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:30:04.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:04.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:04.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:05.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:05.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:05.575 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:30:05.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:30:05.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:05.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:05.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:06.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:06.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:06.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:30:06.871 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:30:06.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:06.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:07.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:08.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:08.280 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:30:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:30:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:09.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:10.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:10.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:10.384 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:30:10.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:30:10.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:10.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:10.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:11.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:11.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:11.836 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:30:11.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:30:11.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:11.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:12.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:13.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:13.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:13.382 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:30:13.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:30:13.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:13.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:13.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:14.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:14.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:14.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:30:14.709 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:30:14.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:14.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:15.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:16.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:16.408 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:30:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:30:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:16.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:16.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:17.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:17.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:17.773 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:30:17.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:30:17.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:17.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:18.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:19.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:19.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:19.510 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:30:19.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:30:19.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:19.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:30:19.817 INFO:tasks.workunit.client.0.smithi067.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/694831278,v1:127.0.0.1:6843/694831278] [v2:127.0.0.1:6844/694831278,v1:127.0.0.1:6845/694831278] exists,up 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:30:19.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:30:19.823 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:30:19.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84b74332-b490-4382-b515-77e4bc421e9f 2024-04-01T05:30:19.825 INFO:tasks.workunit.client.0.smithi067.stdout:add osd6 84b74332-b490-4382-b515-77e4bc421e9f 2024-04-01T05:30:19.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 84b74332-b490-4382-b515-77e4bc421e9f' 2024-04-01T05:30:19.825 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:30:19.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrRgpmjUsFMhAAktAiE4ANWGEu04gCMLCLaw== 2024-04-01T05:30:19.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrRgpmjUsFMhAAktAiE4ANWGEu04gCMLCLaw=="}' 2024-04-01T05:30:19.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84b74332-b490-4382-b515-77e4bc421e9f -i td/osd-scrub-repair/6/new.json 2024-04-01T05:30:20.236 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:30:20.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-04-01T05:30:20.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrRgpmjUsFMhAAktAiE4ANWGEu04gCMLCLaw== --osd-uuid 84b74332-b490-4382-b515-77e4bc421e9f 2024-04-01T05:30:20.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.276+0000 7f6e56d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:20.367 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.368+0000 7f6e56d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:20.368 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.369+0000 7f6e56d45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:20.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.369+0000 7f6e56d45640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-04-01T05:30:20.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.369+0000 7f6e56d45640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-04-01T05:30:20.369 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.369+0000 7f6e56d45640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-04-01T05:30:20.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:20.442+0000 7f6e56d45640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-04-01T05:30:26.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-04-01T05:30:26.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:30:26.685 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd6 key to auth repository 2024-04-01T05:30:26.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-01T05:30:26.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:30:27.052 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.6 2024-04-01T05:30:27.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-01T05:30:27.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:30:27.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:30:27.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:30:27.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:30:27.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:27.079+0000 7f2c69a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:27.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:27.081+0000 7f2c69a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:27.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:27.082+0000 7f2c69a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:27.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:30:27.338 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:30:27.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:27.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:27.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:27.608+0000 7f2c69a14640 -1 Falling back to public interface 2024-04-01T05:30:27.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:28.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:28.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:28.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:30:28.621 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:30:28.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:28.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:28.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:29.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:29.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:29.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:30:29.915 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:30:29.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:29.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:29.937 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:29.938+0000 7f2c69a14640 -1 osd.6 0 log_to_monitors true 2024-04-01T05:30:30.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:31.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:31.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:31.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:30:31.263 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:30:31.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:31.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:31.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:32.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:30:32.684 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:30:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:32.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:32.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:33.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:33.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:33.974 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:30:33.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:30:33.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:33.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:35.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:35.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:35.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:30:35.270 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:30:35.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:35.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:35.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:36.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:36.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:36.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:30:36.589 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:30:36.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:36.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:36.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:37.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:37.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:37.883 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:30:37.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:30:37.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:37.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:38.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:39.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:39.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:39.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:30:39.182 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:30:39.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:39.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:39.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:40.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:40.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:40.516 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:30:40.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:30:40.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:40.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:40.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:41.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:41.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:41.811 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:30:41.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:30:41.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:41.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:42.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:43.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:43.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:43.496 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:30:43.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:30:43.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:43.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:43.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:44.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:44.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:44.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:30:44.789 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:30:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:44.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:45.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:46.636 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:30:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:30:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:46.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:48.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:48.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:48.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:30:48.546 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:30:48.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:48.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:49.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:50.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:50.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:50.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:30:50.249 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:30:50.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:50.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:50.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:51.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:51.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:51.686 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:30:51.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:30:51.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:51.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:52.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:30:53.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:30:53.225 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:30:53.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:30:53.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:30:53.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:30:53.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:30:53.518 INFO:tasks.workunit.client.0.smithi067.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2993778855,v1:127.0.0.1:6851/2993778855] [v2:127.0.0.1:6852/2993778855,v1:127.0.0.1:6853/2993778855] exists,up 84b74332-b490-4382-b515-77e4bc421e9f 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:30:53.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:30:53.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:30:53.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=745d0574-e79f-4d21-910f-95a85365b2f5 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 745d0574-e79f-4d21-910f-95a85365b2f5' 2024-04-01T05:30:53.526 INFO:tasks.workunit.client.0.smithi067.stdout:add osd7 745d0574-e79f-4d21-910f-95a85365b2f5 2024-04-01T05:30:53.527 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:30:53.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNRgpmI7AwIBAArYLe5l6ABXPruGV4BSHsXQ== 2024-04-01T05:30:53.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNRgpmI7AwIBAArYLe5l6ABXPruGV4BSHsXQ=="}' 2024-04-01T05:30:53.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 745d0574-e79f-4d21-910f-95a85365b2f5 -i td/osd-scrub-repair/7/new.json 2024-04-01T05:30:53.869 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:30:53.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-04-01T05:30:53.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNRgpmI7AwIBAArYLe5l6ABXPruGV4BSHsXQ== --osd-uuid 745d0574-e79f-4d21-910f-95a85365b2f5 2024-04-01T05:30:53.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:53.907+0000 7f5a2340f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:53.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:53.945+0000 7f5a2340f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:53.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:53.946+0000 7f5a2340f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:30:53.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:53.946+0000 7f5a2340f640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-04-01T05:30:53.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:53.946+0000 7f5a2340f640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-04-01T05:30:53.947 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:53.946+0000 7f5a2340f640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-04-01T05:30:54.029 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:30:54.028+0000 7f5a2340f640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-04-01T05:30:59.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-04-01T05:30:59.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:30:59.956 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd7 key to auth repository 2024-04-01T05:30:59.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-04-01T05:30:59.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:31:00.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-04-01T05:31:00.328 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.7 2024-04-01T05:31:00.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:31:00.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:31:00.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:31:00.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:31:00.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:00.354+0000 7f7fadc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:00.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:00.356+0000 7f7fadc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:00.356 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:00.357+0000 7f7fadc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:00.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-04-01T05:31:00.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:31:00.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-04-01T05:31:00.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:31:00.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:31:00.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:00.611 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:31:00.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:31:00.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:00.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:00.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:00.880+0000 7f7fadc0d640 -1 Falling back to public interface 2024-04-01T05:31:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:01.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:01.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:01.917 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:31:01.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:31:01.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:01.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:03.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:03.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:03.213 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:31:03.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:31:03.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:03.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:03.232 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:03.233+0000 7f7fadc0d640 -1 osd.7 0 log_to_monitors true 2024-04-01T05:31:03.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:04.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:04.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:04.542 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:31:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:31:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:04.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:04.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:05.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:05.907 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:31:05.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:31:05.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:05.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:06.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:07.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:07.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:07.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:31:07.288 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:31:07.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:07.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:07.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:08.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:08.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:08.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:31:08.585 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:31:08.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:08.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:08.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:09.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:09.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:09.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:31:09.879 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:31:09.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:09.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:10.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:11.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:11.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:11.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:31:11.214 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:31:11.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:11.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:11.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:12.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:12.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:12.558 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:31:12.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:31:12.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:12.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:12.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:13.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:13.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:13.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:31:13.855 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:31:13.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:13.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:14.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:15.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:15.194 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:31:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:31:15.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:15.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:15.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:16.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:16.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:16.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:31:16.625 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:31:16.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:16.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:16.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:17.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:17.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:17.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:31:17.999 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:31:17.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:18.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:18.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:19.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:19.522 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:31:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:31:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:19.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:19.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:20.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:20.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:20.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:31:20.909 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:31:20.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:20.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:21.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:22.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:22.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:22.301 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:31:22.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:31:22.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:22.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:22.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:23.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:31:23.774 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:31:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:23.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:24.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:25.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:25.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:25.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:31:25.471 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:31:25.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:25.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:31:25.889 INFO:tasks.workunit.client.0.smithi067.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1207484537,v1:127.0.0.1:6859/1207484537] [v2:127.0.0.1:6860/1207484537,v1:127.0.0.1:6861/1207484537] exists,up 745d0574-e79f-4d21-910f-95a85365b2f5 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:31:25.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:31:25.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-04-01T05:31:25.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:31:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62553e73-df0f-4825-b10b-4fb3f7447d8d 2024-04-01T05:31:25.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 62553e73-df0f-4825-b10b-4fb3f7447d8d' 2024-04-01T05:31:25.897 INFO:tasks.workunit.client.0.smithi067.stdout:add osd8 62553e73-df0f-4825-b10b-4fb3f7447d8d 2024-04-01T05:31:25.897 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:31:25.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtRgpmoj1PNhAAiJXUdPNKlyScBMtFdcbTaw== 2024-04-01T05:31:25.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtRgpmoj1PNhAAiJXUdPNKlyScBMtFdcbTaw=="}' 2024-04-01T05:31:25.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62553e73-df0f-4825-b10b-4fb3f7447d8d -i td/osd-scrub-repair/8/new.json 2024-04-01T05:31:26.330 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:31:26.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-04-01T05:31:26.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtRgpmoj1PNhAAiJXUdPNKlyScBMtFdcbTaw== --osd-uuid 62553e73-df0f-4825-b10b-4fb3f7447d8d 2024-04-01T05:31:26.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.371+0000 7f31572f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:26.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.382+0000 7f31572f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:26.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.383+0000 7f31572f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:26.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.383+0000 7f31572f9640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-04-01T05:31:26.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.383+0000 7f31572f9640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-04-01T05:31:26.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.383+0000 7f31572f9640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-04-01T05:31:26.483 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:26.483+0000 7f31572f9640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-04-01T05:31:32.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-04-01T05:31:32.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:31:32.734 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd8 key to auth repository 2024-04-01T05:31:32.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-04-01T05:31:32.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:31:33.103 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.8 2024-04-01T05:31:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-04-01T05:31:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:31:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:31:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:31:33.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:31:33.129 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:33.129+0000 7f20c3d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:33.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:33.130+0000 7f20c3d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:33.132 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:33.131+0000 7f20c3d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:33.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:31:33.389 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:31:33.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:33.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:33.654 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:33.654+0000 7f20c3d8c640 -1 Falling back to public interface 2024-04-01T05:31:33.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:34.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:34.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:34.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:31:34.672 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:31:34.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:34.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:34.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:35.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:35.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:35.965 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:31:35.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:31:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:35.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:36.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:31:36.020+0000 7f20c3d8c640 -1 osd.8 0 log_to_monitors true 2024-04-01T05:31:36.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:37.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:37.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:37.261 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:31:37.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:31:37.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:37.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:37.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:38.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:38.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:38.622 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:31:38.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:31:38.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:38.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:38.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:39.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:39.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:39.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:31:39.917 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:31:39.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:39.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:40.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:41.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:41.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:41.210 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:31:41.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:31:41.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:41.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:41.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:42.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:42.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:42.548 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:31:42.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:31:42.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:42.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:42.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:43.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:43.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:43.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:31:43.893 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:31:43.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:43.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:44.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:45.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:45.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:45.187 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:31:45.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:31:45.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:45.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:45.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:46.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:46.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:46.584 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:31:46.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:31:46.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:46.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:46.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:47.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:47.909 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:31:47.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:31:47.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:47.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:48.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:49.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:49.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:49.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:31:49.202 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:31:49.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:49.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:49.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:50.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:50.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:50.564 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:31:50.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:31:50.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:50.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:51.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:52.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:52.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:52.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:31:52.435 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:31:52.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:52.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:52.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:53.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:53.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:53.937 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:31:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:31:53.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:53.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:54.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:55.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:55.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:55.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:31:55.392 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:31:55.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:55.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:55.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:56.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:56.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:56.686 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:31:56.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:31:56.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:56.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:57.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:58.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:58.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:58.413 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:31:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:31:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:58.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:31:58.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:31:59.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:31:59.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:31:59.710 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:31:59.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:31:59.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:31:59.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:32:00.000 INFO:tasks.workunit.client.0.smithi067.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1028754026,v1:127.0.0.1:6867/1028754026] [v2:127.0.0.1:6868/1028754026,v1:127.0.0.1:6869/1028754026] exists,up 62553e73-df0f-4825-b10b-4fb3f7447d8d 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:32:00.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:32:00.006 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:32:00.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:32:00.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:32:00.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 97c51d9e-6334-4f85-be5a-8c7ec186154f' 2024-04-01T05:32:00.008 INFO:tasks.workunit.client.0.smithi067.stdout:add osd9 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:32:00.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:32:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQRgpm5txUARAAzH2bDvGKUyaVHukBJRwU6w== 2024-04-01T05:32:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQRgpm5txUARAAzH2bDvGKUyaVHukBJRwU6w=="}' 2024-04-01T05:32:00.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97c51d9e-6334-4f85-be5a-8c7ec186154f -i td/osd-scrub-repair/9/new.json 2024-04-01T05:32:00.357 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:32:00.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-04-01T05:32:00.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQRgpm5txUARAAzH2bDvGKUyaVHukBJRwU6w== --osd-uuid 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:32:00.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.397+0000 7f2895e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:00.441 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.441+0000 7f2895e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.442+0000 7f2895e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.443+0000 7f2895e0e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-04-01T05:32:00.442 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.443+0000 7f2895e0e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-04-01T05:32:00.443 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.443+0000 7f2895e0e640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-04-01T05:32:00.517 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:00.517+0000 7f2895e0e640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-04-01T05:32:06.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-04-01T05:32:06.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:32:06.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-04-01T05:32:06.550 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd9 key to auth repository 2024-04-01T05:32:06.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:32:06.921 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.9 2024-04-01T05:32:06.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-04-01T05:32:06.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:32:06.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:32:06.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:32:06.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:32:06.948 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:06.949+0000 7fd0c63a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:06.950 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:06.951+0000 7fd0c63a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:06.952+0000 7fd0c63a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:07.208 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:32:07.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:32:07.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:07.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:07.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:07.471+0000 7fd0c63a8640 -1 Falling back to public interface 2024-04-01T05:32:07.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:08.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:08.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:08.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:32:08.502 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:32:08.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:08.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:08.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:09.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:09.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:09.796 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:32:09.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:32:09.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:09.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:09.857 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:09.857+0000 7fd0c63a8640 -1 osd.9 0 log_to_monitors true 2024-04-01T05:32:10.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:11.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:11.089 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:32:11.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:11.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:32:11.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:11.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:11.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:12.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:12.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:12.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:32:12.399 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:32:12.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:12.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:12.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:13.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:13.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:13.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:32:13.824 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:32:13.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:13.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:14.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:15.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:15.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:15.119 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:32:15.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:32:15.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:15.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:15.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:16.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:16.416 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:32:16.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:32:16.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:16.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:16.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:17.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:17.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:17.713 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:32:17.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:32:17.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:17.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:18.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:19.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:19.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:19.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:32:19.080 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:32:19.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:19.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:19.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:20.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:20.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:20.401 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:32:20.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:32:20.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:20.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:20.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:32:21.695 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:32:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:22.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:23.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:23.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:23.083 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:32:23.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:32:23.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:23.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:23.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:24.404 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:32:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:32:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:24.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:24.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:25.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:25.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:25.869 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:32:25.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:32:25.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:25.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:26.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:27.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:27.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:27.408 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:32:27.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:32:27.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:27.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:28.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:29.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:29.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:29.101 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:32:29.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:32:29.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:29.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:29.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:30.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:30.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:30.670 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:32:30.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:32:30.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:30.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:31.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:32.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:32.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:32.013 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:32:32.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:32:32.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:32.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:32.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:32:33.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:32:33.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:33.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:32:33.412 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:32:33.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:33.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:32:33.705 INFO:tasks.workunit.client.0.smithi067.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2924158665,v1:127.0.0.1:6875/2924158665] [v2:127.0.0.1:6876/2924158665,v1:127.0.0.1:6877/2924158665] exists,up 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:32:33.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:32:33.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:32:33.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:32:33.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-04-01T05:32:33.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:32:34.055 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:32:34.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:32:34.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:32:34.508 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:32:34.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:32:35.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:32:38.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-04-01T05:32:38.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:32:38.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:32:38.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:32:38.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:32:38.164 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:32:38.164 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:32:38.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:32:38.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:32:38.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:32:38.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:32:38.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:32:38.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:32:38.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:32:38.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:32:38.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:32:38.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:38.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:32:38.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-04-01T05:32:38.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-04-01T05:32:38.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-04-01T05:32:38.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:38.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:32:38.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673017 2024-04-01T05:32:38.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673017 2024-04-01T05:32:38.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017' 2024-04-01T05:32:38.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:38.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:32:38.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509491 2024-04-01T05:32:38.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509491 2024-04-01T05:32:38.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491' 2024-04-01T05:32:38.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:38.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:32:38.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345967 2024-04-01T05:32:38.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345967 2024-04-01T05:32:38.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967' 2024-04-01T05:32:38.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:38.996 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:32:39.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182439 2024-04-01T05:32:39.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182439 2024-04-01T05:32:39.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967 4-107374182439' 2024-04-01T05:32:39.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:39.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:32:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018912 2024-04-01T05:32:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018912 2024-04-01T05:32:39.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967 4-107374182439 5-128849018912' 2024-04-01T05:32:39.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:39.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:32:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855386 2024-04-01T05:32:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855386 2024-04-01T05:32:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967 4-107374182439 5-128849018912 6-150323855386' 2024-04-01T05:32:39.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:39.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:32:39.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691865 2024-04-01T05:32:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691865 2024-04-01T05:32:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967 4-107374182439 5-128849018912 6-150323855386 7-171798691865' 2024-04-01T05:32:39.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:39.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:32:39.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528334 2024-04-01T05:32:39.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528334 2024-04-01T05:32:39.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967 4-107374182439 5-128849018912 6-150323855386 7-171798691865 8-193273528334' 2024-04-01T05:32:39.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:39.541 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:32:39.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-04-01T05:32:39.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-04-01T05:32:39.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673017 2-64424509491 3-85899345967 4-107374182439 5-128849018912 6-150323855386 7-171798691865 8-193273528334 9-214748364806' 2024-04-01T05:32:39.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:39.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-04-01T05:32:39.647 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:39.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:32:39.648 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-04-01T05:32:39.648 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:39.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-04-01T05:32:39.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-04-01T05:32:39.650 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836543 2024-04-01T05:32:39.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:32:39.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-04-01T05:32:39.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:32:40.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:32:40.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:32:41.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836543 2024-04-01T05:32:41.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:41.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673017 2024-04-01T05:32:41.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:41.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:32:41.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673017 2024-04-01T05:32:41.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:41.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673017 2024-04-01T05:32:41.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2024-04-01T05:32:41.231 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949673017 2024-04-01T05:32:41.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:32:41.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673017 -lt 42949673017 2024-04-01T05:32:41.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:41.531 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509491 2024-04-01T05:32:41.531 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:41.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:32:41.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509491 2024-04-01T05:32:41.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:41.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509491 2024-04-01T05:32:41.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-04-01T05:32:41.534 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509491 2024-04-01T05:32:41.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:32:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509491 -lt 64424509491 2024-04-01T05:32:41.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:41.818 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345967 2024-04-01T05:32:41.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:32:41.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345967 2024-04-01T05:32:41.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:41.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345967 2024-04-01T05:32:41.821 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345967 2024-04-01T05:32:41.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-04-01T05:32:41.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:32:42.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345967 -lt 85899345967 2024-04-01T05:32:42.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:42.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182439 2024-04-01T05:32:42.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:42.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:32:42.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182439 2024-04-01T05:32:42.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:42.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182439 2024-04-01T05:32:42.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2024-04-01T05:32:42.145 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.4 seq 107374182439 2024-04-01T05:32:42.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:32:42.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182439 -lt 107374182439 2024-04-01T05:32:42.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:42.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018912 2024-04-01T05:32:42.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:42.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:32:42.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018912 2024-04-01T05:32:42.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:42.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018912 2024-04-01T05:32:42.537 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.5 seq 128849018912 2024-04-01T05:32:42.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018912' 2024-04-01T05:32:42.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:32:42.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018913 -lt 128849018912 2024-04-01T05:32:42.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:42.818 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855386 2024-04-01T05:32:42.818 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:42.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:32:42.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855386 2024-04-01T05:32:42.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:42.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855386 2024-04-01T05:32:42.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855386' 2024-04-01T05:32:42.821 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.6 seq 150323855386 2024-04-01T05:32:42.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:32:43.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855386 -lt 150323855386 2024-04-01T05:32:43.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:43.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691865 2024-04-01T05:32:43.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:43.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:32:43.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691865 2024-04-01T05:32:43.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691865 2024-04-01T05:32:43.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-04-01T05:32:43.106 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.7 seq 171798691865 2024-04-01T05:32:43.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:32:43.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691866 -lt 171798691865 2024-04-01T05:32:43.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:43.396 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528334 2024-04-01T05:32:43.396 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:43.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:32:43.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528334 2024-04-01T05:32:43.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:43.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528334 2024-04-01T05:32:43.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528334' 2024-04-01T05:32:43.399 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.8 seq 193273528334 2024-04-01T05:32:43.399 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:32:43.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528335 -lt 193273528334 2024-04-01T05:32:43.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:43.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364806 2024-04-01T05:32:43.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:43.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:32:43.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364806 2024-04-01T05:32:43.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:43.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-04-01T05:32:43.688 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.9 seq 214748364806 2024-04-01T05:32:43.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364806' 2024-04-01T05:32:43.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:32:43.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-04-01T05:32:43.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:32:43.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:32:43.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:32:44.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:32:44.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:32:44.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:32:44.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:32:44.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:32:45.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:32:45.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:32:45.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:32:45.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-04-01T05:32:45.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:32:45.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:32:45.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-04-01T05:32:45.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:32:45.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-04-01T05:32:45.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:32:45.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:32:46.473 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:32:46.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:32:47.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-04-01T05:32:47.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:32:47.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:32:47.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:32:47.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:32:47.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:32:47.490 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:32:47.490 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:32:47.490 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:32:47.490 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:32:47.490 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:32:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:32:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:32:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:32:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:32:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:32:47.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:47.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:32:48.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-04-01T05:32:48.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-04-01T05:32:48.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-04-01T05:32:48.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:32:48.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-04-01T05:32:48.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-04-01T05:32:48.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020' 2024-04-01T05:32:48.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:32:48.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509493 2024-04-01T05:32:48.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509493 2024-04-01T05:32:48.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493' 2024-04-01T05:32:48.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:32:48.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345970 2024-04-01T05:32:48.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345970 2024-04-01T05:32:48.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970' 2024-04-01T05:32:48.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:32:48.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182442 2024-04-01T05:32:48.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182442 2024-04-01T05:32:48.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970 4-107374182442' 2024-04-01T05:32:48.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.503 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:32:48.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018915 2024-04-01T05:32:48.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018915 2024-04-01T05:32:48.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970 4-107374182442 5-128849018915' 2024-04-01T05:32:48.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.608 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:32:48.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855389 2024-04-01T05:32:48.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855389 2024-04-01T05:32:48.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970 4-107374182442 5-128849018915 6-150323855389' 2024-04-01T05:32:48.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:32:48.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-04-01T05:32:48.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-04-01T05:32:48.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970 4-107374182442 5-128849018915 6-150323855389 7-171798691868' 2024-04-01T05:32:48.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.827 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:32:48.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528337 2024-04-01T05:32:48.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528337 2024-04-01T05:32:48.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970 4-107374182442 5-128849018915 6-150323855389 7-171798691868 8-193273528337' 2024-04-01T05:32:48.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:32:48.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:32:49.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364809 2024-04-01T05:32:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364809 2024-04-01T05:32:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509493 3-85899345970 4-107374182442 5-128849018915 6-150323855389 7-171798691868 8-193273528337 9-214748364809' 2024-04-01T05:32:49.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:49.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-04-01T05:32:49.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:32:49.037 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-04-01T05:32:49.037 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:49.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-04-01T05:32:49.038 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836546 2024-04-01T05:32:49.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-04-01T05:32:49.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:32:49.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836546 2024-04-01T05:32:49.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:49.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-04-01T05:32:49.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:49.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:32:49.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-04-01T05:32:49.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:49.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-04-01T05:32:49.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-04-01T05:32:49.325 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949673020 2024-04-01T05:32:49.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:32:49.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-04-01T05:32:49.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:49.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509493 2024-04-01T05:32:49.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:49.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:32:49.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509493 2024-04-01T05:32:49.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509493 2024-04-01T05:32:49.622 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509493 2024-04-01T05:32:49.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-04-01T05:32:49.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:32:49.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509494 -lt 64424509493 2024-04-01T05:32:49.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:49.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345970 2024-04-01T05:32:49.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:49.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:32:49.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345970 2024-04-01T05:32:49.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:49.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345970 2024-04-01T05:32:49.945 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345970 2024-04-01T05:32:49.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-04-01T05:32:49.945 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:32:50.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345969 -lt 85899345970 2024-04-01T05:32:50.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:32:51.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:32:51.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:32:51.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345970 -lt 85899345970 2024-04-01T05:32:51.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:51.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182442 2024-04-01T05:32:51.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:51.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:32:51.558 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182442 2024-04-01T05:32:51.558 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:51.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182442 2024-04-01T05:32:51.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-04-01T05:32:51.560 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.4 seq 107374182442 2024-04-01T05:32:51.560 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:32:51.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182442 -lt 107374182442 2024-04-01T05:32:51.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:51.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018915 2024-04-01T05:32:51.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:51.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:32:51.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018915 2024-04-01T05:32:51.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:51.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018915 2024-04-01T05:32:51.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-04-01T05:32:51.850 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.5 seq 128849018915 2024-04-01T05:32:51.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:32:52.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018916 -lt 128849018915 2024-04-01T05:32:52.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:52.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855389 2024-04-01T05:32:52.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:52.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:32:52.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855389 2024-04-01T05:32:52.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:52.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855389 2024-04-01T05:32:52.137 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.6 seq 150323855389 2024-04-01T05:32:52.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2024-04-01T05:32:52.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:32:52.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855389 -lt 150323855389 2024-04-01T05:32:52.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:52.422 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-04-01T05:32:52.422 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:52.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:32:52.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-04-01T05:32:52.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:52.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-04-01T05:32:52.424 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.7 seq 171798691868 2024-04-01T05:32:52.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-04-01T05:32:52.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:32:52.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-04-01T05:32:52.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:52.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528337 2024-04-01T05:32:52.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:52.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:32:52.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528337 2024-04-01T05:32:52.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528337 2024-04-01T05:32:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-04-01T05:32:52.755 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.8 seq 193273528337 2024-04-01T05:32:52.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:32:53.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528338 -lt 193273528337 2024-04-01T05:32:53.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:32:53.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364809 2024-04-01T05:32:53.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:32:53.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:32:53.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364809 2024-04-01T05:32:53.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:32:53.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364809 2024-04-01T05:32:53.041 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.9 seq 214748364809 2024-04-01T05:32:53.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-04-01T05:32:53.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:32:53.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364809 -lt 214748364809 2024-04-01T05:32:53.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:32:53.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:32:53.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:32:53.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:32:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:32:53.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:32:53.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:32:53.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:32:53.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:32:53.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:32:53.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:32:53.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:32:53.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:32:53.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:32:53.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:32:54.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:32:54.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:32:54.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:32:54.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:32:54.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-04-01T05:32:54.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T05:32:54.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-04-01T05:32:54.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-04-01T05:32:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:32:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:32:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-04-01T05:32:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:32:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:32:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:32:54.750 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:32:54.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:32:55.271 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:32:55.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:32:55.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:32:55.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:32:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-04-01T05:32:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:32:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:32:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:32:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:32:56.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-04-01T05:32:56.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-04-01T05:32:56.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T05:32:56.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-04-01T05:32:56.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-01T05:32:56.177 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:32:56.177 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T05:32:56.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-01T05:32:56.504 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:32:56.504 INFO:tasks.workunit.client.0.smithi067.stderr:9 2024-04-01T05:32:56.504 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:32:56.504 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:7' 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-04-01T05:32:56.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:32:56.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:32:56.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:32:56.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:32:58.727 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:32:59.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:32:59.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:32:59.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:32:59.251 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:32:59.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:32:59.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:32:59.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:32:59.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:32:59.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:32:59.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:32:59.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:59.278+0000 7fb723e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:59.281 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:59.281+0000 7fb723e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:59.282 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:59.282+0000 7fb723e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:32:59.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:32:59.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:32:59.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:32:59.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:32:59.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:32:59.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:32:59.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:32:59.550 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:32:59.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:32:59.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:32:59.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:32:59.797+0000 7fb723e11640 -1 Falling back to public interface 2024-04-01T05:32:59.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:00.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:00.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:00.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:33:00.831 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:00.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:00.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:33:01.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:33:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:33:02.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:02.145+0000 7fb723e11640 -1 osd.3 67 log_to_monitors true 2024-04-01T05:33:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:03.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:03.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:03.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:33:03.494 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:03.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:03.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/1967065940,v1:127.0.0.1:6827/1967065940] [v2:127.0.0.1:6828/1967065940,v1:127.0.0.1:6829/1967065940] exists,up d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:33:03.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:33:03.875 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:33:03.875 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:33:03.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:33:03.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:33:03.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:33:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:33:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:33:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:33:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:33:03.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:33:03.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:33:04.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:33:04.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-04-01T05:33:04.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-04-01T05:33:04.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-04-01T05:33:04.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:33:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673024 2024-04-01T05:33:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673024 2024-04-01T05:33:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024' 2024-04-01T05:33:04.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:33:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-04-01T05:33:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-04-01T05:33:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498' 2024-04-01T05:33:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:33:04.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-04-01T05:33:04.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-04-01T05:33:04.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018' 2024-04-01T05:33:04.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.765 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:33:04.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182446 2024-04-01T05:33:04.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182446 2024-04-01T05:33:04.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018 4-107374182446' 2024-04-01T05:33:04.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:33:04.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018919 2024-04-01T05:33:04.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018919 2024-04-01T05:33:04.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018 4-107374182446 5-128849018919' 2024-04-01T05:33:04.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:04.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:33:05.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855393 2024-04-01T05:33:05.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855393 2024-04-01T05:33:05.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018 4-107374182446 5-128849018919 6-150323855393' 2024-04-01T05:33:05.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:05.095 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:33:05.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691872 2024-04-01T05:33:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691872 2024-04-01T05:33:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018 4-107374182446 5-128849018919 6-150323855393 7-171798691872' 2024-04-01T05:33:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:05.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:33:05.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528342 2024-04-01T05:33:05.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528342 2024-04-01T05:33:05.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018 4-107374182446 5-128849018919 6-150323855393 7-171798691872 8-193273528342' 2024-04-01T05:33:05.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:05.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:33:05.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364813 2024-04-01T05:33:05.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364813 2024-04-01T05:33:05.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673024 2-64424509498 3-304942678018 4-107374182446 5-128849018919 6-150323855393 7-171798691872 8-193273528342 9-214748364813' 2024-04-01T05:33:05.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:05.453 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-04-01T05:33:05.453 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:05.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:33:05.454 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-04-01T05:33:05.454 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-04-01T05:33:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-04-01T05:33:05.455 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836551 2024-04-01T05:33:05.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:33:05.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836551 2024-04-01T05:33:05.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:33:06.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:33:06.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:33:07.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-04-01T05:33:07.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:07.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673024 2024-04-01T05:33:07.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:07.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:33:07.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673024 2024-04-01T05:33:07.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:07.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673024 2024-04-01T05:33:07.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-04-01T05:33:07.042 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673024 2024-04-01T05:33:07.042 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:33:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673024 -lt 42949673024 2024-04-01T05:33:07.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:07.354 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-04-01T05:33:07.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:07.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:33:07.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-04-01T05:33:07.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:07.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-04-01T05:33:07.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-04-01T05:33:07.357 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509498 2024-04-01T05:33:07.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:33:07.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-04-01T05:33:07.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:07.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-04-01T05:33:07.643 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:07.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:33:07.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-04-01T05:33:07.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:07.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-04-01T05:33:07.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-04-01T05:33:07.645 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 304942678018 2024-04-01T05:33:07.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:33:07.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-04-01T05:33:07.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:07.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182446 2024-04-01T05:33:07.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:07.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:33:07.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182446 2024-04-01T05:33:07.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:07.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182446 2024-04-01T05:33:07.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-04-01T05:33:07.929 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182446 2024-04-01T05:33:07.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:33:08.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182446 -lt 107374182446 2024-04-01T05:33:08.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:08.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018919 2024-04-01T05:33:08.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:08.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:33:08.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018919 2024-04-01T05:33:08.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:08.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018919 2024-04-01T05:33:08.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018919' 2024-04-01T05:33:08.217 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 128849018919 2024-04-01T05:33:08.217 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:33:08.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018920 -lt 128849018919 2024-04-01T05:33:08.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:08.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855393 2024-04-01T05:33:08.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:08.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:33:08.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855393 2024-04-01T05:33:08.503 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855393 2024-04-01T05:33:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855393' 2024-04-01T05:33:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855393 2024-04-01T05:33:08.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:33:08.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855393 -lt 150323855393 2024-04-01T05:33:08.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:08.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691872 2024-04-01T05:33:08.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:08.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:33:08.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691872 2024-04-01T05:33:08.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:08.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691872 2024-04-01T05:33:08.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2024-04-01T05:33:08.794 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691872 2024-04-01T05:33:08.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:33:09.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691873 -lt 171798691872 2024-04-01T05:33:09.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:09.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528342 2024-04-01T05:33:09.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:09.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:33:09.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528342 2024-04-01T05:33:09.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:09.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528342 2024-04-01T05:33:09.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528342' 2024-04-01T05:33:09.079 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528342 2024-04-01T05:33:09.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:33:09.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528342 -lt 193273528342 2024-04-01T05:33:09.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:09.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364813 2024-04-01T05:33:09.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:09.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:33:09.363 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364813 2024-04-01T05:33:09.363 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:09.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364813 2024-04-01T05:33:09.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-04-01T05:33:09.364 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 214748364813 2024-04-01T05:33:09.365 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:33:09.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364813 -lt 214748364813 2024-04-01T05:33:09.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:33:09.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:09.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:33:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:33:10.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:33:10.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:33:10.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:10.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:10.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:33:10.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:33:11.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:33:11.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:33:11.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:33:11.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:33:11.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:33:11.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:33:11.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:33:11.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:33:11.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:32:46.273637+0000 2024-04-01T05:33:11.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:33:11.558 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:32:46.273637+0000 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:32:46.273637+0000 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:33:11.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:33:11.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:33:11.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:33:11.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:32:46.273637+0000 '>' 2024-04-01T05:32:46.273637+0000 2024-04-01T05:33:11.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:33:12.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:33:12.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:33:12.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:33:12.851 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:33:12.852 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:33:12.852 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:33:12.852 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:33:13.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:12.353480+0000 '>' 2024-04-01T05:32:46.273637+0000 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:33:13.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:33:13.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:33:13.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:33:13.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:33:13.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:33:13.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:33:13.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:33:13.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:33:13.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:33:13.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:33:13.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:33:13.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-04-01T05:33:15.599 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:33:15.599 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:33:15.599 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:33:15.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:33:15.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:33:15.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:33:15.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:33:15.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:33:15.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:33:15.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:33:15.877 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.9 2024-04-01T05:33:15.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:33:15.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:33:15.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:33:15.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:33:15.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:33:15.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:33:15.905 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:15.905+0000 7fa3797bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:15.907 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:15.907+0000 7fa3797bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:15.908 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:15.909+0000 7fa3797bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:16.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:33:16.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:33:16.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:16.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:16.423 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:16.424+0000 7fa3797bb640 -1 Falling back to public interface 2024-04-01T05:33:16.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:33:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:17.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:18.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:18.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:18.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:33:18.764 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:18.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:18.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:18.858 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:18.859+0000 7fa3797bb640 -1 osd.9 72 log_to_monitors true 2024-04-01T05:33:19.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:33:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:20.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/3038047129,v1:127.0.0.1:6875/3038047129] [v2:127.0.0.1:6876/3038047129,v1:127.0.0.1:6877/3038047129] exists,up 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:33:20.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:33:20.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:33:20.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:33:20.452 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:33:20.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:33:20.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:33:20.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:33:20.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:33:20.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:33:20.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:33:21.181 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:33:21.182 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:33:21.182 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:33:21.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:33:21.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.182 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:33:21.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836555 2024-04-01T05:33:21.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836555 2024-04-01T05:33:21.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555' 2024-04-01T05:33:21.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:33:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673028 2024-04-01T05:33:21.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673028 2024-04-01T05:33:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028' 2024-04-01T05:33:21.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:33:21.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509502 2024-04-01T05:33:21.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509502 2024-04-01T05:33:21.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502' 2024-04-01T05:33:21.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:33:21.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678022 2024-04-01T05:33:21.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678022 2024-04-01T05:33:21.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022' 2024-04-01T05:33:21.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:33:21.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182451 2024-04-01T05:33:21.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182451 2024-04-01T05:33:21.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022 4-107374182451' 2024-04-01T05:33:21.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:33:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018924 2024-04-01T05:33:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018924 2024-04-01T05:33:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022 4-107374182451 5-128849018924' 2024-04-01T05:33:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.878 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:33:21.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-04-01T05:33:21.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-04-01T05:33:21.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022 4-107374182451 5-128849018924 6-150323855397' 2024-04-01T05:33:21.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:21.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:33:22.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691877 2024-04-01T05:33:22.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691877 2024-04-01T05:33:22.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022 4-107374182451 5-128849018924 6-150323855397 7-171798691877' 2024-04-01T05:33:22.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:22.089 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:33:22.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528346 2024-04-01T05:33:22.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528346 2024-04-01T05:33:22.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022 4-107374182451 5-128849018924 6-150323855397 7-171798691877 8-193273528346' 2024-04-01T05:33:22.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:22.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:33:22.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-04-01T05:33:22.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-04-01T05:33:22.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673028 2-64424509502 3-304942678022 4-107374182451 5-128849018924 6-150323855397 7-171798691877 8-193273528346 9-330712481794' 2024-04-01T05:33:22.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:22.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836555 2024-04-01T05:33:22.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:22.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:33:22.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836555 2024-04-01T05:33:22.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:22.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836555 2024-04-01T05:33:22.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-04-01T05:33:22.303 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836555 2024-04-01T05:33:22.303 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:33:22.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836555 2024-04-01T05:33:22.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:22.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673028 2024-04-01T05:33:22.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:22.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:33:22.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673028 2024-04-01T05:33:22.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:22.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673028 2024-04-01T05:33:22.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-04-01T05:33:22.622 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673028 2024-04-01T05:33:22.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:33:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673029 -lt 42949673028 2024-04-01T05:33:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:22.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509502 2024-04-01T05:33:22.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:22.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:33:22.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509502 2024-04-01T05:33:22.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:22.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509502 2024-04-01T05:33:22.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-04-01T05:33:22.909 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509502 2024-04-01T05:33:22.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:33:23.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509502 2024-04-01T05:33:23.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:23.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678022 2024-04-01T05:33:23.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:23.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:33:23.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678022 2024-04-01T05:33:23.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:23.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678022 2024-04-01T05:33:23.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678022' 2024-04-01T05:33:23.198 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 304942678022 2024-04-01T05:33:23.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:33:23.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678022 2024-04-01T05:33:23.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:23.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182451 2024-04-01T05:33:23.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:23.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:33:23.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182451 2024-04-01T05:33:23.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:23.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182451 2024-04-01T05:33:23.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-04-01T05:33:23.500 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182451 2024-04-01T05:33:23.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:33:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182451 2024-04-01T05:33:23.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:23.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018924 2024-04-01T05:33:23.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:23.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:33:23.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018924 2024-04-01T05:33:23.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:23.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018924 2024-04-01T05:33:23.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-04-01T05:33:23.788 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 128849018924 2024-04-01T05:33:23.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:33:24.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018924 2024-04-01T05:33:24.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:24.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-04-01T05:33:24.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:24.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:33:24.115 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-04-01T05:33:24.115 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:24.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-04-01T05:33:24.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-04-01T05:33:24.116 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855397 2024-04-01T05:33:24.116 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:33:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855397 2024-04-01T05:33:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:24.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691877 2024-04-01T05:33:24.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:24.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:33:24.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691877 2024-04-01T05:33:24.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691877 2024-04-01T05:33:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-04-01T05:33:24.404 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691877 2024-04-01T05:33:24.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:33:24.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691877 -lt 171798691877 2024-04-01T05:33:24.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:24.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528346 2024-04-01T05:33:24.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:33:24.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528346 2024-04-01T05:33:24.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:24.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528346 2024-04-01T05:33:24.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-04-01T05:33:24.703 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528346 2024-04-01T05:33:24.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:33:24.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528346 2024-04-01T05:33:24.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:24.987 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-04-01T05:33:24.987 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:24.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:33:24.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-04-01T05:33:24.989 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:24.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-04-01T05:33:24.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-04-01T05:33:24.990 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 330712481794 2024-04-01T05:33:24.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:33:25.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-04-01T05:33:25.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:33:25.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:25.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:33:25.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:33:25.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:33:25.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:33:25.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:33:25.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:33:25.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:33:25.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:33:25.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:33:25.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:33:25.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:25.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:26.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:33:26.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:33:26.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:33:26.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:33:26.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:33:26.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-04-01T05:33:26.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:33:26.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:33:26.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:33:26.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:33:26.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:33:26.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:33:26.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-04-01T05:33:28.680 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:33:29.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:33:29.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:33:29.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:33:29.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:33:29.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:33:29.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:33:29.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:33:29.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:33:29.200 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.5 2024-04-01T05:33:29.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:33:29.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:33:29.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:33:29.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:33:29.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:33:29.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:33:29.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:29.227+0000 7fe1b6a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:29.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:29.230+0000 7fe1b6a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:29.231 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:29.231+0000 7fe1b6a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:33:29.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:29.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:33:29.756 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:29.757+0000 7fe1b6a15640 -1 Falling back to public interface 2024-04-01T05:33:29.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:33:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:33:31.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:32.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:32.092+0000 7fe1b6a15640 -1 osd.5 79 log_to_monitors true 2024-04-01T05:33:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:32.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:33:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:32.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:33:32.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:33:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/1496129059,v1:127.0.0.1:6843/1496129059] [v2:127.0.0.1:6844/1496129059,v1:127.0.0.1:6845/1496129059] exists,up 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:33:33.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:33:33.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:33:33.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:33:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:33:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:33:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:33:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:33:33.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:33:33.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:33:34.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:34.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:33:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-04-01T05:33:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-04-01T05:33:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-04-01T05:33:34.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:34.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:33:34.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-04-01T05:33:34.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-04-01T05:33:34.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032' 2024-04-01T05:33:34.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:34.869 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:33:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-04-01T05:33:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-04-01T05:33:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506' 2024-04-01T05:33:34.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:34.974 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:33:35.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678026 2024-04-01T05:33:35.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678026 2024-04-01T05:33:35.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026' 2024-04-01T05:33:35.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:35.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:33:35.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182454 2024-04-01T05:33:35.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182454 2024-04-01T05:33:35.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026 4-107374182454' 2024-04-01T05:33:35.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:35.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:33:35.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-04-01T05:33:35.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-04-01T05:33:35.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026 4-107374182454 5-360777252866' 2024-04-01T05:33:35.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:35.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:33:35.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855401 2024-04-01T05:33:35.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855401 2024-04-01T05:33:35.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026 4-107374182454 5-360777252866 6-150323855401' 2024-04-01T05:33:35.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:35.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:33:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691880 2024-04-01T05:33:35.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691880 2024-04-01T05:33:35.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026 4-107374182454 5-360777252866 6-150323855401 7-171798691880' 2024-04-01T05:33:35.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:35.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:33:35.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528350 2024-04-01T05:33:35.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528350 2024-04-01T05:33:35.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026 4-107374182454 5-360777252866 6-150323855401 7-171798691880 8-193273528350' 2024-04-01T05:33:35.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:35.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:33:35.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-04-01T05:33:35.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-04-01T05:33:35.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673032 2-64424509506 3-304942678026 4-107374182454 5-360777252866 6-150323855401 7-171798691880 8-193273528350 9-330712481798' 2024-04-01T05:33:35.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:35.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-04-01T05:33:35.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:35.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:33:35.713 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-04-01T05:33:35.713 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:35.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-04-01T05:33:35.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-04-01T05:33:35.715 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836559 2024-04-01T05:33:35.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:33:35.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836559 2024-04-01T05:33:35.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:33:36.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:33:36.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:33:37.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836559 2024-04-01T05:33:37.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:37.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-04-01T05:33:37.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:37.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:33:37.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-04-01T05:33:37.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-04-01T05:33:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-04-01T05:33:37.296 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673032 2024-04-01T05:33:37.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:33:37.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-04-01T05:33:37.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:37.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-04-01T05:33:37.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:37.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:33:37.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-04-01T05:33:37.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:37.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-04-01T05:33:37.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-04-01T05:33:37.582 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509506 2024-04-01T05:33:37.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:33:37.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509506 2024-04-01T05:33:37.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:37.864 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678026 2024-04-01T05:33:37.864 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:37.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:33:37.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678026 2024-04-01T05:33:37.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:37.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678026 2024-04-01T05:33:37.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678026' 2024-04-01T05:33:37.867 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 304942678026 2024-04-01T05:33:37.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:33:38.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678026 -lt 304942678026 2024-04-01T05:33:38.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:38.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182454 2024-04-01T05:33:38.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:38.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:33:38.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182454 2024-04-01T05:33:38.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:38.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182454 2024-04-01T05:33:38.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-04-01T05:33:38.157 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182454 2024-04-01T05:33:38.157 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:33:38.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182454 2024-04-01T05:33:38.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:38.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-04-01T05:33:38.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:38.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:33:38.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-04-01T05:33:38.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:38.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-04-01T05:33:38.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-04-01T05:33:38.462 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 360777252866 2024-04-01T05:33:38.462 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:33:38.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-04-01T05:33:38.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:38.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855401 2024-04-01T05:33:38.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:38.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:33:38.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855401 2024-04-01T05:33:38.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:38.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855401 2024-04-01T05:33:38.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-04-01T05:33:38.750 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855401 2024-04-01T05:33:38.750 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:33:39.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855401 2024-04-01T05:33:39.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691880 2024-04-01T05:33:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:39.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:33:39.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691880 2024-04-01T05:33:39.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:39.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691880 2024-04-01T05:33:39.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691880' 2024-04-01T05:33:39.040 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691880 2024-04-01T05:33:39.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:33:39.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691881 -lt 171798691880 2024-04-01T05:33:39.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:39.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528350 2024-04-01T05:33:39.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:39.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:33:39.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528350 2024-04-01T05:33:39.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528350 2024-04-01T05:33:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-04-01T05:33:39.324 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528350 2024-04-01T05:33:39.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:33:39.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528350 -lt 193273528350 2024-04-01T05:33:39.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:39.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-04-01T05:33:39.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:39.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:33:39.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-04-01T05:33:39.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:39.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-04-01T05:33:39.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-04-01T05:33:39.628 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 330712481798 2024-04-01T05:33:39.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:33:39.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-04-01T05:33:39.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:33:39.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:39.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:33:40.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:33:40.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:33:40.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:33:40.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:40.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:40.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:33:40.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:33:40.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:33:40.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:33:40.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:33:40.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:33:40.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:33:40.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:33:41.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:33:41.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:33:12.353480+0000 2024-04-01T05:33:41.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:33:41.780 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:33:12.353480+0000 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:33:12.353480+0000 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:33:41.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:33:41.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:33:41.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:33:41.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:33:41.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:33:42.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:12.353480+0000 '>' 2024-04-01T05:33:12.353480+0000 2024-04-01T05:33:42.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:33:43.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:33:43.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:33:43.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:33:43.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:33:43.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:33:43.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:33:43.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:42.316806+0000 '>' 2024-04-01T05:33:12.353480+0000 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:33:43.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:33:43.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:33:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:33:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:33:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:33:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:33:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:33:43.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:33:43.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:33:43.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-04-01T05:33:45.676 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:33:45.676 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:33:45.676 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:33:45.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:33:45.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:33:45.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:33:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:33:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.9 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:33:45.953 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:33:45.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:33:45.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:33:45.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:33:45.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:33:45.980 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:45.979+0000 7fba4bc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:45.982 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:45.982+0000 7fba4bc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:45.983 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:45.983+0000 7fba4bc13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:33:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:33:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:33:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:33:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:33:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:33:46.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:46.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:33:46.246 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:33:46.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:46.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:46.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:46.500+0000 7fba4bc13640 -1 Falling back to public interface 2024-04-01T05:33:46.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:47.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:47.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:47.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:33:47.589 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:47.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:47.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:47.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:33:48.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:33:48.825+0000 7fba4bc13640 -1 osd.9 86 log_to_monitors true 2024-04-01T05:33:48.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:33:48.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:33:48.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:33:48.884 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:48.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:33:48.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:33:49.305 INFO:tasks.workunit.client.0.smithi067.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [77,86) [v2:127.0.0.1:6874/3372323311,v1:127.0.0.1:6875/3372323311] [v2:127.0.0.1:6876/3372323311,v1:127.0.0.1:6877/3372323311] exists,up 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:33:49.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:33:49.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:33:49.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:33:49.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:33:49.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:33:49.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:33:49.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:33:49.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:33:49.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:33:49.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:33:49.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:33:49.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:33:49.788 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:33:49.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:33:49.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:49.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:33:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-04-01T05:33:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-04-01T05:33:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-04-01T05:33:49.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:49.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:33:50.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-04-01T05:33:50.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-04-01T05:33:50.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036' 2024-04-01T05:33:50.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:50.128 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:33:50.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-04-01T05:33:50.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-04-01T05:33:50.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510' 2024-04-01T05:33:50.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:50.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:33:50.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678030 2024-04-01T05:33:50.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678030 2024-04-01T05:33:50.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030' 2024-04-01T05:33:50.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:50.482 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:33:50.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-04-01T05:33:50.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-04-01T05:33:50.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030 4-107374182458' 2024-04-01T05:33:50.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:50.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:33:50.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252870 2024-04-01T05:33:50.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252870 2024-04-01T05:33:50.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030 4-107374182458 5-360777252870' 2024-04-01T05:33:50.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:50.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:33:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855405 2024-04-01T05:33:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855405 2024-04-01T05:33:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030 4-107374182458 5-360777252870 6-150323855405' 2024-04-01T05:33:51.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:51.074 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:33:51.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691884 2024-04-01T05:33:51.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-04-01T05:33:51.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030 4-107374182458 5-360777252870 6-150323855405 7-171798691884' 2024-04-01T05:33:51.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:51.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:33:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-04-01T05:33:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-04-01T05:33:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030 4-107374182458 5-360777252870 6-150323855405 7-171798691884 8-193273528354' 2024-04-01T05:33:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:33:51.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:33:51.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-04-01T05:33:51.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-04-01T05:33:51.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509510 3-304942678030 4-107374182458 5-360777252870 6-150323855405 7-171798691884 8-193273528354 9-386547056642' 2024-04-01T05:33:51.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:51.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-04-01T05:33:51.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:51.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:33:51.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-04-01T05:33:51.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:51.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-04-01T05:33:51.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-04-01T05:33:51.395 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836563 2024-04-01T05:33:51.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:33:51.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836563 2024-04-01T05:33:51.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:51.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-04-01T05:33:51.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:51.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:33:51.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-04-01T05:33:51.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:51.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-04-01T05:33:51.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-04-01T05:33:51.682 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673036 2024-04-01T05:33:51.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:33:51.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673036 2024-04-01T05:33:51.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:51.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-04-01T05:33:51.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:51.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:33:51.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-04-01T05:33:51.971 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:51.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-04-01T05:33:51.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-04-01T05:33:51.972 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509510 2024-04-01T05:33:51.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:33:52.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-04-01T05:33:52.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:52.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678030 2024-04-01T05:33:52.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:52.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:33:52.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678030 2024-04-01T05:33:52.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:52.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678030 2024-04-01T05:33:52.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678030' 2024-04-01T05:33:52.256 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 304942678030 2024-04-01T05:33:52.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:33:52.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678031 -lt 304942678030 2024-04-01T05:33:52.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:52.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-04-01T05:33:52.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:52.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:33:52.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-04-01T05:33:52.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:52.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-04-01T05:33:52.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-04-01T05:33:52.545 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182458 2024-04-01T05:33:52.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:33:52.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182459 -lt 107374182458 2024-04-01T05:33:52.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:52.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252870 2024-04-01T05:33:52.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:52.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:33:52.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252870 2024-04-01T05:33:52.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:52.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252870 2024-04-01T05:33:52.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252870' 2024-04-01T05:33:52.846 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 360777252870 2024-04-01T05:33:52.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:33:53.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252870 -lt 360777252870 2024-04-01T05:33:53.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:53.172 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855405 2024-04-01T05:33:53.172 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:53.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:33:53.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855405 2024-04-01T05:33:53.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:53.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855405 2024-04-01T05:33:53.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-04-01T05:33:53.174 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855405 2024-04-01T05:33:53.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:33:53.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855405 -lt 150323855405 2024-04-01T05:33:53.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:53.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-04-01T05:33:53.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:53.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:33:53.470 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-04-01T05:33:53.470 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:53.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-04-01T05:33:53.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-04-01T05:33:53.471 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691884 2024-04-01T05:33:53.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:33:53.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691885 -lt 171798691884 2024-04-01T05:33:53.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:53.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-04-01T05:33:53.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:53.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:33:53.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-04-01T05:33:53.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:53.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-04-01T05:33:53.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-04-01T05:33:53.756 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528354 2024-04-01T05:33:53.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:33:54.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528354 -lt 193273528354 2024-04-01T05:33:54.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:33:54.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-386547056642 2024-04-01T05:33:54.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:33:54.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:33:54.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-386547056642 2024-04-01T05:33:54.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:33:54.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-04-01T05:33:54.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-04-01T05:33:54.042 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 386547056642 2024-04-01T05:33:54.043 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:33:54.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-04-01T05:33:54.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:33:54.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:54.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:54.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:33:54.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:33:54.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:33:54.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:33:54.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:33:54.683 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:33:54.683 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:33:54.683 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:33:54.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:33:54.962 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:33:54.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:33:54.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:33:55.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:33:55.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:33:55.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:33:55.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:33:55.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:33:55.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-04-01T05:33:55.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:33:55.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229377"' 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229377' 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229379"' 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229379' 2024-04-01T05:33:55.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229377 229379' 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229377 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229380: /' 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:33:55.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229382: /' 2024-04-01T05:33:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:33:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:33:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:33:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:58.399 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:33:58.400 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:33:58.401 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:33:58.402 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:33:58.402 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:33:58.402 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:33:58.402 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:33:58.402 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:33:58.402 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:33:58.403 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: start osd.5 2024-04-01T05:33:58.404 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:33:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:33:58.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: remove 2#2:eb822e21:::SOMETHING:head# 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:33:58.436 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:33:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:33:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:33:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:33:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:33:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:33:58.437 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:33:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: start osd.9 2024-04-01T05:33:58.439 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2024-04-01T05:33:58.432+0000 7f58d7619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:03.225 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2024-04-01T05:33:58.448+0000 7f58d7619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2024-04-01T05:33:58.449+0000 7f58d7619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 0 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2024-04-01T05:33:58.962+0000 7f58d7619640 -1 Falling back to public interface 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 1 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.227 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2024-04-01T05:34:01.618+0000 7f58d7619640 -1 osd.5 91 log_to_monitors true 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 3 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229380: osd.5 up in weight 1 up_from 96 up_thru 84 down_at 92 last_229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:03.228 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2024-04-01T05:33:58.467+0000 7ff289215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2024-04-01T05:33:58.469+0000 7ff289215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2024-04-01T05:33:58.471+0000 7ff289215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 0 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.229 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2024-04-01T05:33:58.987+0000 7ff289215640 -1 Falling back to public interface 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 1 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2 2024-04-01T05:34:03.230 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2024-04-01T05:34:01.817+0000 7ff289215640 -1 osd.9 91 log_to_monitors true 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 3 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:03.231 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: osd.9 up in weight 1 up_from 97 up_thru 0 down_at 92 last_clean_interval [90,91) [v2:127.0.0.1:6874/621605716,v1:127.0.0.1:6875/621605716] [v2:127.0.0.1:6876/621605716,v1:127.0.0.1:6877/621605716] exists,up 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:34:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 1 2024-04-01T05:34:04.326 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 2 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 3 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 4 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 5 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 6 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 7 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 8 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: 9' 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:04.327 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041' 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/staclean_interval [84,91) [v2:127.0.0.1:6842/3920219059,v1:127.0.0.1:6843/3920219059] [v2:127.0.0.1:6844/3920219059,v1:127.0.0.1:6845/3920219059] exists,up 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:34:04.328 INFO:tasks.workunit.client.0.smithi067.stderr:229380: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 1 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 2 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 3 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 4 2024-04-01T05:34:04.329 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 5 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 6 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 7 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 8 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: 9' 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-04-01T05:34:04.330 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040' 2024-04-01T05:34:04.331 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.741 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:34:05.741 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-04-01T05:34:05.741 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514' 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678034 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678034 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034' 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034 4-107374182463' 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.742 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034 4-107374182463 5-412316860418' 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034 4-107374182463 5-412316860418 6-150323855410' 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691889 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-04-01T05:34:05.743 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034 4-107374182463 5-412316860418 6-150323855410 7-171798691889' 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clonndalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515' 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678035 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678035 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035' 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:34:05.744 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035 4-107374182462' 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035 4-107374182462 5-412316860419' 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-04-01T05:34:05.745 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035 4-107374182462 5-412316860419 6-150323855409' 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691888 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691888 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035 4-107374182462 5-412316860419 6-150323855409 7-171798691888' 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:05.746 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:34:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-04-01T05:34:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-04-01T05:34:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035 4-107374182462 5-412316860419 6-150323855409 7-171798691888 8-193273528359' 2024-04-01T05:34:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:06.433 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673041 2-64424509515 3-304942678035 4-107374182462 5-412316860419 6-150323855409 7-171798691888 8-193273528359 9-416611827715' 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.0 seq 21474836567 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836567 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:06.434 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.1 seq 42949673041 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:06.435 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.cliene.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034 4-107374182463 5-412316860418 6-150323855410 7-171798691889 8-193273528358' 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673040 2-64424509514 3-304942678034 4-107374182463 5-412316860418 6-150323855410 7-171798691889 8-193273528358 9-416611827714' 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:06.485 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.0 seq 21474836568 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:34:06.486 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.1 seq 42949673040 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673040 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:06.487 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.t.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.2 seq 64424509515 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509515 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:07.348 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678035 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678035 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678035 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678035' 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.3 seq 304942678035 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678035 -lt 304942678035 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.349 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.4 seq 107374182462 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182462 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860419 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860419 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.350 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting oclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.2 seq 64424509514 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509514 2024-04-01T05:34:07.451 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678034 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678034 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678034 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678034' 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.3 seq 304942678034 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678035 -lt 304942678034 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.452 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.4 seq 107374182463 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182463 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860418 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860418 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:07.453 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitsd.5 seq 412316860419' 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.5 seq 412316860419 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.6 seq 150323855409 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855409 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691888 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691888 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691888 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.7 seq 171798691888 2024-04-01T05:34:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691889 -lt 171798691888 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.8 seq 193273528359 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:34:08.545 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528359 2024-04-01T05:34:08.753 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_staing osd.5 seq 412316860418' 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.5 seq 412316860418 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.6 seq 150323855410 2024-04-01T05:34:08.754 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855410 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.7 seq 171798691889 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691889 -lt 171798691889 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:08.755 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.8 seq 193273528358 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:34:08.756 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528358 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pts: for s in $seqs 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-416611827715 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-416611827715 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 416611827715' 2024-04-01T05:34:10.066 INFO:tasks.workunit.client.0.smithi067.stderr:229382: waiting osd.9 seq 416611827715 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827716 -lt 416611827715 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:34:10.067 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:10.068 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:10.242 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/g_stats: for s in $seqs 2024-04-01T05:34:10.242 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-416611827714 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-416611827714 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 416611827714' 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: waiting osd.9 seq 416611827714 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827716 -lt 416611827714 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:10.243 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:34:10.244 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:34:10.245 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:10.245 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:10.935 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:10.935 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:10.935 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:10.935 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:10.935 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:10.935 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:229380: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:229380: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:34:10.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229379 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:34:11.281 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:34:11.282 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:11.283 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:11.967 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:11.967 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:34:11.967 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:11.967 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:229382: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:229382: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:34:11.968 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:34:11.969 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:34:12.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:34:12.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:34:12.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:34:12.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:34:12.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:12.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:12.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:12.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:12.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:12.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:34:12.807 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:34:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:34:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:34:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:34:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:12.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:12.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:12.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:12.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:12.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:13.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:42.316806+0000 '>' 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:13.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:14.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:14.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:14.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:14.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:14.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:14.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:14.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:14.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:42.316806+0000 '>' 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:14.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:15.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:15.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:15.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:15.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:15.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:15.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:15.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:15.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:42.316806+0000 '>' 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:15.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:16.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:16.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:16.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:16.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:16.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:16.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:16.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:16.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:42.316806+0000 '>' 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:16.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:17.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:17.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:17.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:17.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:17.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:17.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:17.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:18.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:33:42.316806+0000 '>' 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:18.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:19.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:19.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:19.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:19.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:19.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:19.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:19.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:19.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:34:13.387773+0000 '>' 2024-04-01T05:33:42.316806+0000 2024-04-01T05:34:19.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232981"' 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232981' 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 232982"' 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 232982' 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 232981 232982' 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:34:19.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:19.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:34:19.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232981 2024-04-01T05:34:19.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232984: /' 2024-04-01T05:34:19.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:34:19.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/232986: /' 2024-04-01T05:34:22.222 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.222 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:34:22.222 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:34:22.222 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:34:22.222 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:34:22.222 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:34:22.223 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: _ 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: hinfo_key 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: snapset 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:34:22.224 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:34:22.225 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:34:22.226 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:34:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:34:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:34:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:34:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:34:22.227 INFO:tasks.workunit.client.0.smithi067.stderr:232986: start osd.9 2024-04-01T05:34:22.364 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.364 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:34:22.364 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:34:22.365 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-04-01T05:34:22.366 INFO:tasks.workunit.client.0.smithi067.stderr:232984: _ 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: hinfo_key 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: snapset 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:34:22.367 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:34:22.368 INFO:tasks.workunit.client.0.smithi067.stderr:232984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:34:22.369 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:34:22.370 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:34:22.370 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:34:22.370 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:34:22.370 INFO:tasks.workunit.client.0.smithi067.stderr:232984: start osd.5 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2024-04-01T05:34:22.255+0000 7effc2bfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2024-04-01T05:34:22.257+0000 7effc2bfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2024-04-01T05:34:22.258+0000 7effc2bfa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 0 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2024-04-01T05:34:22.788+0000 7effc2bfa640 -1 Falling back to public interface 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 1 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2024-04-01T05:34:25.372+0000 7effc2bfa640 -1 osd.9 99 log_to_monitors true 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:26.819 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:26.820 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:26.820 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 3 2024-04-01T05:34:26.820 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:26.820 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232986: osd.9 up in weight 1 up_from 103 up_thru 0 down_aovery_ops 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2024-04-01T05:34:22.394+0000 7f11d6813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2024-04-01T05:34:22.396+0000 7f11d6813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2024-04-01T05:34:22.397+0000 7f11d6813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:34:27.171 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 0 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2024-04-01T05:34:22.909+0000 7f11d6813640 -1 Falling back to public interface 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 1 2024-04-01T05:34:27.172 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2024-04-01T05:34:25.749+0000 7f11d6813640 -1 osd.5 100 log_to_monitors true 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 3 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:27.173 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:27.769 INFO:tasks.workunit.client.0.smithi067.stderr:232984: osd.5 up in weight 1 up_from 104 up_thru 96 downt 100 last_clean_interval [97,99) [v2:127.0.0.1:6842/4082913489,v1:127.0.0.1:6843/4082913489] [v2:127.0.0.1:6844/4082913489,v1:127.0.0.1:6845/4082913489] exists,up 97c51d9e-6334-4f85-be5a-8c7ec186154f 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:34:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:232986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 1 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 2 2024-04-01T05:34:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 3 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 4 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 5 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 6 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 7 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 8 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: 9' 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-04-01T05:34:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047' 2024-04-01T05:34:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone_at 100 last_clean_interval [96,100) [v2:127.0.0.1:6874/1781006974,v1:127.0.0.1:6875/1781006974] [v2:127.0.0.1:6876/1781006974,v1:127.0.0.1:6877/1781006974] exists,up 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:34:28.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 1 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 2 2024-04-01T05:34:28.341 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 3 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 4 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 5 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 6 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 7 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 8 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: 9' 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:34:28.342 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-04-01T05:34:28.343 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-04-01T05:34:28.343 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048' 2024-04-01T05:34:28.343 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.389 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/cl.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:34:29.389 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-04-01T05:34:29.389 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-04-01T05:34:29.389 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520' 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678041 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678041 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041' 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041 4-107374182469' 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598786 2024-04-01T05:34:29.390 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598786 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041 4-107374182469 5-446676598786' 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855416 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855416 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041 4-107374182469 5-446676598786 6-150323855416' 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691895 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041 4-107374182469 5-446676598786 6-150323855416 7-171798691895' 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.391 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:34:29.608 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/ceone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522' 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678042 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678042 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042' 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182470 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182470 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042 4-107374182470' 2024-04-01T05:34:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=446676598787 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 446676598787 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042 4-107374182470 5-446676598787' 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042 4-107374182470 5-446676598787 6-150323855417' 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691896 2024-04-01T05:34:29.610 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691896 2024-04-01T05:34:29.611 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042 4-107374182470 5-446676598787 6-150323855417 7-171798691896' 2024-04-01T05:34:29.611 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:29.611 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntuphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041 4-107374182469 5-446676598786 6-150323855416 7-171798691895 8-193273528364' 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-04-01T05:34:30.103 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673047 2-64424509520 3-304942678041 4-107374182469 5-446676598786 6-150323855416 7-171798691895 8-193273528364 9-442381631490' 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.0 seq 21474836573 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836573 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:34:30.104 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.1 seq 42949673047 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673047 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:30.105 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/ceph/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042 4-107374182470 5-446676598787 6-150323855417 7-171798691896 8-193273528366' 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673048 2-64424509522 3-304942678042 4-107374182470 5-446676598787 6-150323855417 7-171798691896 8-193273528366 9-442381631491' 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-04-01T05:34:30.338 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.0 seq 21474836574 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836574 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:30.339 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.1 seq 42949673048 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673048 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:30.340 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/ctest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.2 seq 64424509520 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509520 2024-04-01T05:34:31.084 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678041 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678041 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678041 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678041' 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.3 seq 304942678041 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678042 -lt 304942678041 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:34:31.085 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.4 seq 107374182469 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182469 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598786 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598786 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.086 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598786 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.2 seq 64424509522 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:31.235 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509522 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678042 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678042 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678042 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.3 seq 304942678042 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678042 -lt 304942678042 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182470 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:31.236 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182470 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182470 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.4 seq 107374182470 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182470 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-446676598787 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-446676598787 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:31.237 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=446676598787 2024-04-01T05:34:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stat echo 'waiting osd.5 seq 446676598786' 2024-04-01T05:34:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.5 seq 446676598786 2024-04-01T05:34:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:34:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598786 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855416 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855416 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855416 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.6 seq 150323855416 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855416 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:32.330 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.7 seq 171798691895 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691895 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-04-01T05:34:32.331 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-04-01T05:34:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.8 seq 193273528364 2024-04-01T05:34:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:34:32.332 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528364 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22s: echo 'waiting osd.5 seq 446676598787' 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.5 seq 446676598787 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 446676598787 -lt 446676598787 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-04-01T05:34:32.504 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.6 seq 150323855417 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855417 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691896 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691896 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691896 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691896' 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.7 seq 171798691896 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691896 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:32.505 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.8 seq 193273528366 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:34:32.506 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528366 2024-04-01T05:34:33.706 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh10: flush_pg_stats: for s in $seqs 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631490 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631490 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631490' 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: waiting osd.9 seq 442381631490 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631490 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:33.707 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:232986: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:34:33.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:34:33.853 INFO:tasks.workunit.client.0.smithi067.stderr::2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:33.853 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631491 2024-04-01T05:34:33.853 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:33.853 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631491 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631491' 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: waiting osd.9 seq 442381631491 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631492 -lt 442381631491 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:33.854 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:232984: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 232982 2024-04-01T05:34:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:34:33.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:34:33.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:34:33.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:34:33.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:34:33.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:34:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:34:33.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236214"' 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236214' 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236216"' 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236216' 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 236214 236216' 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:34:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:34:33.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236214 2024-04-01T05:34:33.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236217: /' 2024-04-01T05:34:33.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:34:33.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236219: /' 2024-04-01T05:34:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:34:36.792 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:34:36.793 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:34:36.794 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:34:36.795 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:34:36.795 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:34:36.795 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:34:36.795 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:34:36.795 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:34:36.795 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:34:36.796 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:34:36.797 INFO:tasks.workunit.client.0.smithi067.stderr:236219: start osd.5 2024-04-01T05:34:36.798 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:34:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:34:36.959 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:34:36.960 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:34:36.961 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:34:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:34:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:34:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:34:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:34:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:34:36.962 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: start osd.3 2024-04-01T05:34:36.964 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:34:41.642 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:34:41.642 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:34:41.642 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:41.642 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:41.642 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:41.642 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2024-04-01T05:34:36.988+0000 7f1ecf8a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2024-04-01T05:34:36.990+0000 7f1ecf8a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2024-04-01T05:34:36.991+0000 7f1ecf8a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 0 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2024-04-01T05:34:37.505+0000 7f1ecf8a0640 -1 Falling back to public interface 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:41.643 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 1 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:41.644 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2024-04-01T05:34:40.255+0000 7f1ecf8a0640 -1 osd.3 107 log_to_monitors true 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 3 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236217: osd.3 up in weight 1 up_from 113 up_thru 113 down_at 108 l236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2024-04-01T05:34:36.825+0000 7febba96f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2024-04-01T05:34:36.827+0000 7febba96f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2024-04-01T05:34:36.828+0000 7febba96f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:34:41.645 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 0 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2024-04-01T05:34:37.344+0000 7febba96f640 -1 Falling back to public interface 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:41.646 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 1 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2024-04-01T05:34:40.011+0000 7febba96f640 -1 osd.5 107 log_to_monitors true 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:41.647 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 3 2024-04-01T05:34:41.648 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:41.648 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: osd.5 up in weight 1 up_from 113 up_thru 113 down_at 108 last_clean_interval [104,107) [v2:127.0.0.1:6826/2508995751,v1:127.0.0.1:6827/2508995751] [v2:127.0.0.1:6828/2508995751,v1:127.0.0.1:6829/2508995751] exists,up 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:34:42.349 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 1 2024-04-01T05:34:42.350 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 2 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 3 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 4 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 5 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 6 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 7 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 8 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: 9' 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:34:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052' 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.clieast_clean_interval [71,107) [v2:127.0.0.1:6874/1455520756,v1:127.0.0.1:6875/1455520756] [v2:127.0.0.1:6876/1455520756,v1:127.0.0.1:6877/1455520756] exists,up d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:34:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:236217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 1 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 2 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 3 2024-04-01T05:34:42.353 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 4 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 5 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 6 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 7 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 8 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: 9' 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:34:42.354 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-04-01T05:34:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-04-01T05:34:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053' 2024-04-01T05:34:42.355 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.220 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:34:43.220 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-04-01T05:34:43.220 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-04-01T05:34:43.220 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526' 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451' 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451 4-107374182474' 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-04-01T05:34:43.221 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451 4-107374182474 5-485331304450' 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451 4-107374182474 5-485331304450 6-150323855421' 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451 4-107374182474 5-485331304450 6-150323855421 7-171798691900' 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtestnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:34:43.222 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525' 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450' 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182475 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182475 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450 4-107374182475' 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:34:43.223 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450 4-107374182475 5-485331304451' 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450 4-107374182475 5-485331304451 6-150323855420' 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691901 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691901 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450 4-107374182475 5-485331304451 6-150323855420 7-171798691901' 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.224 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:34:43.940 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtes/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-04-01T05:34:43.940 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451 4-107374182474 5-485331304450 6-150323855421 7-171798691900 8-193273528369' 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631496 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631496 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673053 2-64424509526 3-485331304451 4-107374182474 5-485331304450 6-150323855421 7-171798691900 8-193273528369 9-442381631496' 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-04-01T05:34:43.941 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.0 seq 21474836578 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836578 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.1 seq 42949673053 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:34:43.942 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673053 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/ct/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450 4-107374182475 5-485331304451 6-150323855420 7-171798691901 8-193273528370' 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:34:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631495 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631495 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673052 2-64424509525 3-485331304450 4-107374182475 5-485331304451 6-150323855420 7-171798691901 8-193273528370 9-442381631495' 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.0 seq 21474836579 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.1 seq 42949673052 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:34:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673052 2024-04-01T05:34:43.946 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:43.946 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-04-01T05:34:43.946 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:43.946 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:34:46.305 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.2 seq 64424509525 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509525 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509525 2024-04-01T05:34:46.306 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-485331304450 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-485331304450 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 485331304450' 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.3 seq 485331304450 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182475 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:34:46.307 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182475 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182475 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.4 seq 107374182475 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182475 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304451 2024-04-01T05:34:46.308 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:46.313 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2lone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-04-01T05:34:46.313 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.2 seq 64424509526 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509526 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:34:46.314 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509526 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-485331304451 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:34:46.315 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-485331304451 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 485331304451' 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.3 seq 485331304451 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304451 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-04-01T05:34:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.4 seq 107374182474 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:34:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182474 2024-04-01T05:34:46.318 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:46.318 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304450 2024-04-01T05:34:46.318 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22211: flush_pg_stats: osd=5 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304451 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 485331304451' 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.5 seq 485331304451 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:34:47.325 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304452 -lt 485331304451 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.6 seq 150323855420 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855420 2024-04-01T05:34:47.326 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691901 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691901 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691901 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.7 seq 171798691901 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691901 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-04-01T05:34:47.327 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.328 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-04-01T05:34:47.376 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.c11: flush_pg_stats: osd=5 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304450 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 485331304450' 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.5 seq 485331304450 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304452 -lt 485331304450 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-04-01T05:34:47.377 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.6 seq 150323855421 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855421 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-04-01T05:34:47.378 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.7 seq 171798691900 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691900 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:47.379 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-04-01T05:34:48.956 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.cllient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.8 seq 193273528370 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528370 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631495 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631495 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631495 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631495' 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: waiting osd.9 seq 442381631495 2024-04-01T05:34:48.957 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631497 -lt 442381631495 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:48.958 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:236219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:236219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:34:48.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:34:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:ient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-04-01T05:34:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.8 seq 193273528369 2024-04-01T05:34:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:34:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528369 2024-04-01T05:34:49.044 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631496 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631496 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631496 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631496' 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: waiting osd.9 seq 442381631496 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631497 -lt 442381631496 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:49.045 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:34:49.046 INFO:tasks.workunit.client.0.smithi067.stderr:236217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236216 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:34:49.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:49.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:49.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:34:13.387773+0000 2024-04-01T05:34:49.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:34:49.881 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:34:13.387773+0000 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:34:13.387773+0000 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:49.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:50.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:34:13.387773+0000 '>' 2024-04-01T05:34:13.387773+0000 2024-04-01T05:34:50.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:51.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:51.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:51.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:51.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:51.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:51.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:34:13.387773+0000 '>' 2024-04-01T05:34:13.387773+0000 2024-04-01T05:34:51.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:34:52.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:34:52.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:34:52.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:34:52.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:34:52.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:34:52.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:34:52.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:34:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:34:50.233831+0000 '>' 2024-04-01T05:34:13.387773+0000 2024-04-01T05:34:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:34:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:34:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239616"' 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239616' 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239617"' 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239617' 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 239616 239617' 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:34:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239616 2024-04-01T05:34:52.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:34:52.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239619: /' 2024-04-01T05:34:52.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:52.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239621: /' 2024-04-01T05:34:55.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:34:55.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:34:55.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:34:55.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:34:55.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:34:55.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:34:55.541 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: _ 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: hinfo_key 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: snapset 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:34:55.542 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:34:55.543 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:55.544 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:34:55.545 INFO:tasks.workunit.client.0.smithi067.stderr:239619: start osd.3 2024-04-01T05:34:56.004 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:34:56.004 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:34:56.004 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:34:56.004 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:34:56.004 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:56.004 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:34:56.005 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: _ 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: hinfo_key 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: snapset 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:34:56.006 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:34:56.007 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:34:56.008 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:34:56.009 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:34:56.009 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:34:56.009 INFO:tasks.workunit.client.0.smithi067.stderr:239621: start osd.5 2024-04-01T05:34:59.853 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:34:59.853 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:34:59.853 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2024-04-01T05:34:55.572+0000 7f03c5bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2024-04-01T05:34:55.575+0000 7f03c5bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2024-04-01T05:34:55.576+0000 7f03c5bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 0 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2024-04-01T05:34:56.096+0000 7f03c5bbd640 -1 Falling back to public interface 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 1 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:59.855 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2024-04-01T05:34:58.544+0000 7f03c5bbd640 -1 osd.3 114 log_to_monitors true 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 3 2024-04-01T05:34:59.856 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:00.598 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-04-01T05:35:00.598 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2024-04-01T05:34:56.033+0000 7f29e53ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2024-04-01T05:34:56.035+0000 7f29e53ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2024-04-01T05:34:56.036+0000 7f29e53ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:35:00.599 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 0 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2024-04-01T05:34:56.552+0000 7f29e53ac640 -1 Falling back to public interface 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:35:00.600 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 1 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2024-04-01T05:34:59.475+0000 7f29e53ac640 -1 osd.5 114 log_to_monitors true 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:35:00.601 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 3 2024-04-01T05:35:00.602 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:00.602 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:35:01.391 INFO:tasks.workunit.client.0.smithi067.stderr:239621: osd.5 up in weight 1 up_from 120 up_thru 113 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:35:01.391 INFO:tasks.workunit.client.0.smithi067.stderr:239619: osd.3 up in weight 1 up_from 118 up_thru 113 down_at 115 last_clean_interval [113,114) [v2:127.0.0.1:6826/765079372,v1:127.0.0.1:6827/765079372] [v2:127.0.0.1:6828/765079372,v1:127.0.0.1:6829/765079372] exists,up d27cca05-2607-451e-8a01-4ff3f81a18c8 2024-04-01T05:35:01.391 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:35:01.391 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:35:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 1 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 2 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 3 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 4 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 5 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 6 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 7 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 8 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: 9' 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:35:01.393 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-04-01T05:35:01.394 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057' 2024-04-01T05:35:01.540 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/n_at 115 last_clean_interval [113,114) [v2:127.0.0.1:6874/169003976,v1:127.0.0.1:6875/169003976] [v2:127.0.0.1:6876/169003976,v1:127.0.0.1:6877/169003976] exists,up 755f7163-e86b-437f-a28a-2f0e44cec5c4 2024-04-01T05:35:01.540 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:35:01.540 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:35:01.540 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:35:01.540 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:35:01.540 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:35:01.541 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 1 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 2 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 3 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 4 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 5 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 6 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 7 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 8 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: 9' 2024-04-01T05:35:01.542 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059' 2024-04-01T05:35:01.543 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clonstandalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531' 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930' 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:35:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930 4-107374182480' 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930 4-107374182480 5-515396075522' 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855426 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855426 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930 4-107374182480 5-515396075522 6-150323855426' 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:35:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691906 2024-04-01T05:35:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691906 2024-04-01T05:35:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930 4-107374182480 5-515396075522 6-150323855426 7-171798691906' 2024-04-01T05:35:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/ste.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:35:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-04-01T05:35:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532' 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931' 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931 4-107374182481' 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931 4-107374182481 5-515396075523' 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931 4-107374182481 5-515396075523 6-150323855427' 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691907 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691907 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931 4-107374182481 5-515396075523 6-150323855427 7-171798691907' 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:35:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/candalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:35:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-04-01T05:35:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-04-01T05:35:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930 4-107374182480 5-515396075522 6-150323855426 7-171798691906 8-193273528375' 2024-04-01T05:35:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631501 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631501 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509531 3-506806140930 4-107374182480 5-515396075522 6-150323855426 7-171798691906 8-193273528375 9-442381631501' 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.0 seq 21474836584 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:35:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836584 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.1 seq 42949673057 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673057 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-04-01T05:35:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtesephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931 4-107374182481 5-515396075523 6-150323855427 7-171798691907 8-193273528376' 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631502 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631502 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509532 3-506806140931 4-107374182481 5-515396075523 6-150323855427 7-171798691907 8-193273528376 9-442381631502' 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:03.191 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.0 seq 21474836585 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836585 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-04-01T05:35:03.192 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.1 seq 42949673059 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:03.193 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cept/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.2 seq 64424509531 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509531 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:04.030 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140930 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140930 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140930' 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.3 seq 506806140930 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140930 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-04-01T05:35:04.031 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.4 seq 107374182480 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182480 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075522 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075522 2024-04-01T05:35:04.032 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.112 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607htest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.2 seq 64424509532 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140931 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140931 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-04-01T05:35:04.113 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140931' 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.3 seq 506806140931 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140931 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.4 seq 107374182481 2024-04-01T05:35:04.114 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182481 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075523 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075523 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:04.115 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-04-01T05:35:05.242 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats:5522 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075522' 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.5 seq 515396075522 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075522 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855426 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855426 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855426 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-04-01T05:35:05.243 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.6 seq 150323855426 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855426 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691906 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691906 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691906 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691906' 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.7 seq 171798691906 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:35:05.244 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691907 -lt 171798691906 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.8 seq 193273528375 2024-04-01T05:35:05.245 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 19327 echo 'waiting osd.5 seq 515396075523' 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.5 seq 515396075523 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075523 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-04-01T05:35:05.323 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.6 seq 150323855427 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855427 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691907 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691907 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:05.324 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691907 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.7 seq 171798691907 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691907 -lt 171798691907 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.8 seq 193273528376 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:35:05.325 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528376 -lt 193273528376 2024-04-01T05:35:06.612 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:23528376 -lt 193273528375 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631501 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631501 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631501 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631501' 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: waiting osd.9 seq 442381631501 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631501 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:35:06.613 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:35:06.614 INFO:tasks.workunit.client.0.smithi067.stderr:239619: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:35:06.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:35:06.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:35:06.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239617 2024-04-01T05:35:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:210: flush_pg_stats: for s in $seqs 2024-04-01T05:35:06.688 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631502 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631502 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631502 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631502' 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: waiting osd.9 seq 442381631502 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631502 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:35:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:35:06.690 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:239621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:35:06.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:35:06.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:35:06.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:35:06.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:35:06.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:35:06.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:35:06.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:35:06.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:35:06.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:35:06.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:35:06.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:35:06.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:35:06.884 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:35:06.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:35:06.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:35:06.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:35:06.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:35:06.886 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:35:06.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:35:06.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:35:06.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:35:06.888 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:35:06.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:35:06.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:35:06.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:35:06.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:35:06.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:06.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:06.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:35:06.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:35:06.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:35:06.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:35:06.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:35:06.953 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:35:06.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:35:06.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:35:06.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:35:06.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:35:06.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:35:06.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:35:06.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:35:06.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:35:06.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:35:06.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:35:06.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:35:06.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:06.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:06.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:35:06.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:35:06.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:35:06.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:35:06.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:35:06.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:06.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:06.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:35:06.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:35:06.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:35:06.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:35:06.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:35:07.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:35:07.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:35:07.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:35:07.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:35:07.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:35:07.530 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:35:07.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:35:07.616 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:07.617 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:07.617 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:35:07.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:35:07.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:35:07.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:35:08.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:35:08.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:35:08.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:35:08.506 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:35:08.506 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:08.506 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:08.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:35:08.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:35:08.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:35:08.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:35:08.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:35:08.540 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:35:08.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:35:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:35:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:35:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:35:08.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:35:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28fc20a3-e7f8-46f3-b947-0427f95c8d60 2024-04-01T05:35:08.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 28fc20a3-e7f8-46f3-b947-0427f95c8d60' 2024-04-01T05:35:08.543 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 28fc20a3-e7f8-46f3-b947-0427f95c8d60 2024-04-01T05:35:08.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:35:08.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMRwpm8n5MIRAAuUhPSeDgCrjIkQSePLA4NA== 2024-04-01T05:35:08.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMRwpm8n5MIRAAuUhPSeDgCrjIkQSePLA4NA=="}' 2024-04-01T05:35:08.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28fc20a3-e7f8-46f3-b947-0427f95c8d60 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:35:08.811 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:35:08.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:35:08.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMRwpm8n5MIRAAuUhPSeDgCrjIkQSePLA4NA== --osd-uuid 28fc20a3-e7f8-46f3-b947-0427f95c8d60 2024-04-01T05:35:08.850 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.850+0000 7f3b63c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:08.862 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.861+0000 7f3b63c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:08.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.863+0000 7f3b63c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:08.863 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.863+0000 7f3b63c22640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:35:08.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.863+0000 7f3b63c22640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:35:08.864 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.863+0000 7f3b63c22640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:35:08.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:08.945+0000 7f3b63c22640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:35:14.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:35:14.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:35:14.488 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:35:14.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:35:14.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:35:14.865 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:35:14.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:35:14.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:35:14.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:35:14.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:35:14.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:35:14.891 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:14.891+0000 7f71d21e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:14.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:14.903+0000 7f71d21e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:14.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:14.904+0000 7f71d21e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:15.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:35:15.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:35:15.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:35:15.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:35:15.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:35:15.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:15.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:35:15.152 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:35:15.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:15.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:15.421 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:15.420+0000 7f71d21e1640 -1 Falling back to public interface 2024-04-01T05:35:15.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:16.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:16.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:16.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:35:16.435 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:35:16.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:16.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:16.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:17.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:17.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:17.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:35:17.726 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:35:17.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:17.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:17.754 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:17.753+0000 7f71d21e1640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:35:18.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:19.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:19.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:19.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:35:19.199 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:35:19.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:19.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:19.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:20.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:20.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:20.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:35:20.553 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:35:20.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:20.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:20.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:21.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:21.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:21.848 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:35:21.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:35:21.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:21.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:22.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:23.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:23.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:23.142 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:35:23.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:35:23.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:23.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:23.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:24.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:24.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:24.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:35:24.486 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:35:24.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:24.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:24.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:25.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:25.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:25.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:35:25.781 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:35:25.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:25.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:26.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:27.073 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:35:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:35:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:27.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:28.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:28.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:28.477 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:35:28.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:35:28.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:28.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:28.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:29.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:29.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:29.803 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:35:29.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:35:29.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:29.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:30.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:31.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:31.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:31.245 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:35:31.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:35:31.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:31.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:32.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:33.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:33.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:33.023 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:35:33.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:35:33.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:33.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:33.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:34.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:34.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:34.583 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:35:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:35:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:34.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:35.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:36.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:36.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:36.328 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:35:36.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:35:36.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:36.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:36.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:37.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:37.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:37.988 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:35:37.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:35:37.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:37.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:39.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:39.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:39.500 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:35:39.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:35:39.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:39.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:35:39.823 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2949684082,v1:127.0.0.1:6803/2949684082] [v2:127.0.0.1:6804/2949684082,v1:127.0.0.1:6805/2949684082] exists,up 28fc20a3-e7f8-46f3-b947-0427f95c8d60 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:35:39.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:35:39.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:35:39.829 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:35:39.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6694c11-1854-4702-99a6-ff0524110790 2024-04-01T05:35:39.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c6694c11-1854-4702-99a6-ff0524110790' 2024-04-01T05:35:39.830 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 c6694c11-1854-4702-99a6-ff0524110790 2024-04-01T05:35:39.830 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:35:39.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrRwpmWu9VMhAAihE4BGErWETqIzKMsnqjbQ== 2024-04-01T05:35:39.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrRwpmWu9VMhAAihE4BGErWETqIzKMsnqjbQ=="}' 2024-04-01T05:35:39.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6694c11-1854-4702-99a6-ff0524110790 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:35:40.210 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:35:40.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:35:40.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrRwpmWu9VMhAAihE4BGErWETqIzKMsnqjbQ== --osd-uuid c6694c11-1854-4702-99a6-ff0524110790 2024-04-01T05:35:40.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.250+0000 7fd1ae1fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.286+0000 7fd1ae1fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:40.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.287+0000 7fd1ae1fa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.287+0000 7fd1ae1fa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:35:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.287+0000 7fd1ae1fa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:35:40.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.288+0000 7fd1ae1fa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:35:40.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:40.353+0000 7fd1ae1fa640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:35:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:35:46.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:35:46.075 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:35:46.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:35:46.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:35:46.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:35:46.454 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:35:46.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:35:46.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:35:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:35:46.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:35:46.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:46.477+0000 7f939f017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:46.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:46.479+0000 7f939f017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:46.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:46.480+0000 7f939f017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:35:46.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:35:46.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:35:46.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:35:46.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:35:46.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:35:46.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:46.754 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:35:46.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:35:46.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:46.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:46.998 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:46.999+0000 7f939f017640 -1 Falling back to public interface 2024-04-01T05:35:47.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:48.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:48.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:48.036 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:35:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:35:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:48.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:49.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:49.330 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:35:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:35:49.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:49.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:49.342 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:35:49.343+0000 7f939f017640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:35:49.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:50.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:50.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:50.652 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:35:50.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:35:50.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:50.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:50.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:51.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:35:51.966 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:35:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:51.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:52.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:53.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:53.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:53.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:35:53.259 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:35:53.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:53.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:53.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:54.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:54.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:54.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:35:54.554 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:35:54.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:54.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:54.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:55.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:55.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:55.847 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:35:55.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:35:55.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:55.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:56.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:57.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:57.222 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:35:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:35:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:57.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:57.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:58.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:58.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:58.518 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:35:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:35:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:35:58.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:35:59.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:35:59.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:35:59.814 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:35:59.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:35:59.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:35:59.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:00.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:01.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:01.205 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:36:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:36:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:01.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:01.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:02.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:02.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:02.760 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:36:02.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:36:02.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:02.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:03.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:04.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:04.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:04.495 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:36:04.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:36:04.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:04.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:05.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:06.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:06.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:06.265 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:36:06.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:36:06.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:06.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:06.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:07.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:07.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:07.847 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:36:07.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:36:07.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:07.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:08.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:09.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:09.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:09.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:36:09.480 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:36:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:09.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:09.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:10.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:10.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:10.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:36:10.897 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:36:10.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:10.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:11.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:12.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:12.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:12.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:36:12.591 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:36:12.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:12.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:36:12.888 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/435598029,v1:127.0.0.1:6811/435598029] [v2:127.0.0.1:6812/435598029,v1:127.0.0.1:6813/435598029] exists,up c6694c11-1854-4702-99a6-ff0524110790 2024-04-01T05:36:12.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:36:12.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:36:12.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:36:12.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:36:12.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:36:12.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:36:12.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb8eb4e9-88ab-4525-b484-9ec72d682325 2024-04-01T05:36:12.895 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 cb8eb4e9-88ab-4525-b484-9ec72d682325 2024-04-01T05:36:12.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cb8eb4e9-88ab-4525-b484-9ec72d682325' 2024-04-01T05:36:12.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:36:12.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMRwpm+NNNNhAAt3dQTnkJIdurjrt+2JIN2g== 2024-04-01T05:36:12.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMRwpm+NNNNhAAt3dQTnkJIdurjrt+2JIN2g=="}' 2024-04-01T05:36:12.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb8eb4e9-88ab-4525-b484-9ec72d682325 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:36:13.252 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:36:13.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:36:13.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMRwpm+NNNNhAAt3dQTnkJIdurjrt+2JIN2g== --osd-uuid cb8eb4e9-88ab-4525-b484-9ec72d682325 2024-04-01T05:36:13.292 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.293+0000 7f4de8a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:13.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.328+0000 7f4de8a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.329+0000 7f4de8a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.330+0000 7f4de8a15640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:36:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.330+0000 7f4de8a15640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:36:13.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.330+0000 7f4de8a15640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:36:13.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:13.445+0000 7f4de8a15640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:36:19.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:36:19.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:36:19.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:36:19.481 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:36:19.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:36:19.851 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:36:19.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:36:19.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:36:19.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:36:19.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:36:19.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:36:19.877 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:19.878+0000 7fc0a580e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:19.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:19.880+0000 7fc0a580e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:19.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:19.881+0000 7fc0a580e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:20.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:36:20.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:36:20.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:36:20.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:36:20.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:36:20.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:20.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:36:20.134 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:36:20.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:20.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:20.397 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:20.398+0000 7fc0a580e640 -1 Falling back to public interface 2024-04-01T05:36:20.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:21.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:21.415 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:36:21.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:21.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:36:21.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:21.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:21.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:22.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:22.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:22.711 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:36:22.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:36:22.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:22.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:22.786 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:22.786+0000 7fc0a580e640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:36:23.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:24.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:24.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:24.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:36:24.003 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:36:24.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:24.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:24.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:25.323 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:36:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:36:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:25.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:26.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:26.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:26.620 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:36:26.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:36:26.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:26.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:26.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:27.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:27.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:27.914 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:36:27.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:36:27.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:27.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:28.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:29.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:29.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:29.211 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:36:29.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:36:29.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:29.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:29.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:30.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:30.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:30.620 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:36:30.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:36:30.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:30.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:30.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:31.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:31.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:31.914 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:36:31.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:36:31.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:31.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:32.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:33.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:33.208 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:36:33.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:36:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:33.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:33.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:34.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:34.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:34.652 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:36:34.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:36:34.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:34.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:35.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:36.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:36.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:36.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:36:36.095 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:36:36.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:36.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:36.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:37.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:37.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:37.653 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:36:37.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:36:37.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:37.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:38.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:39.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:39.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:39.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:36:39.470 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:36:39.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:39.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:39.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:40.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:40.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:40.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:36:40.806 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:36:40.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:40.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:41.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:42.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:42.568 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:36:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:36:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:42.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:43.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:44.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:44.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:44.018 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:36:44.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:36:44.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:44.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:44.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:45.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:45.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:45.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:36:45.698 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:36:45.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:45.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:36:45.991 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1021072567,v1:127.0.0.1:6819/1021072567] [v2:127.0.0.1:6820/1021072567,v1:127.0.0.1:6821/1021072567] exists,up cb8eb4e9-88ab-4525-b484-9ec72d682325 2024-04-01T05:36:45.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:36:45.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:36:45.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:36:45.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:36:45.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:36:45.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f106e71b-2872-44b6-86db-49f229aa28e9' 2024-04-01T05:36:45.999 INFO:tasks.workunit.client.0.smithi067.stdout:add osd3 f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:36:45.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:36:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuRwpmtT/LABAAgcjcSwqHJ0jnWmd60Z8Paw== 2024-04-01T05:36:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuRwpmtT/LABAAgcjcSwqHJ0jnWmd60Z8Paw=="}' 2024-04-01T05:36:46.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f106e71b-2872-44b6-86db-49f229aa28e9 -i td/osd-scrub-repair/3/new.json 2024-04-01T05:36:46.394 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:36:46.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-04-01T05:36:46.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuRwpmtT/LABAAgcjcSwqHJ0jnWmd60Z8Paw== --osd-uuid f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:36:46.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.433+0000 7f9e40222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:46.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.477+0000 7f9e40222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:46.479 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.478+0000 7f9e40222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:46.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.479+0000 7f9e40222640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:36:46.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.479+0000 7f9e40222640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:36:46.480 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.479+0000 7f9e40222640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-04-01T05:36:46.536 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:46.536+0000 7f9e40222640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-04-01T05:36:52.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-04-01T05:36:52.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:36:52.540 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd3 key to auth repository 2024-04-01T05:36:52.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-01T05:36:52.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:36:52.917 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.3 2024-04-01T05:36:52.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-01T05:36:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:36:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:36:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:36:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:36:52.943 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:52.943+0000 7f19a9214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:52.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:52.945+0000 7f19a9214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:52.946 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:52.946+0000 7f19a9214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:53.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:36:53.201 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:36:53.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:53.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:36:53.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:53.466+0000 7f19a9214640 -1 Falling back to public interface 2024-04-01T05:36:53.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:54.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:54.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:54.488 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:36:54.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:36:54.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:54.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:36:54.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:55.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:55.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:55.782 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:36:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:36:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:55.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:36:55.873 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:36:55.873+0000 7f19a9214640 -1 osd.3 0 log_to_monitors true 2024-04-01T05:36:56.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:57.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:57.074 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:36:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:36:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:57.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:36:57.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:58.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:58.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:58.372 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:36:58.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:36:58.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:58.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:36:58.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:36:59.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:36:59.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:36:59.668 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:36:59.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:36:59.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:36:59.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:36:59.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:00.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:00.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:00.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:37:00.965 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:37:00.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:00.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:01.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:02.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:02.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:02.261 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:37:02.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:37:02.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:02.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:03.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:03.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:03.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:37:03.637 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:37:03.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:03.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:03.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:04.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:04.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:04.976 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:37:04.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:37:04.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:04.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:06.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:06.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:06.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:37:06.271 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:37:06.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:06.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:06.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:07.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:07.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:07.699 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:37:07.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:37:07.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:07.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:08.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:09.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:09.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:09.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:37:09.063 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:37:09.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:09.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:10.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:11.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:11.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:11.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:37:11.046 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:37:11.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:11.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:11.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:12.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:12.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:12.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:37:12.824 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:37:12.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:12.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:13.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:14.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:14.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:14.473 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:37:14.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:37:14.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:14.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:15.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:16.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:16.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:16.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:37:16.112 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:37:16.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:16.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:16.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:17.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:17.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:17.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:37:17.591 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:37:17.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:17.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:17.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:18.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:18.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:18.983 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:37:18.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:37:18.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:18.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:19.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:20.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:20.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:20.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:37:20.280 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:37:20.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:20.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:37:20.574 INFO:tasks.workunit.client.0.smithi067.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3673483584,v1:127.0.0.1:6827/3673483584] [v2:127.0.0.1:6828/3673483584,v1:127.0.0.1:6829/3673483584] exists,up f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:37:20.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:37:20.579 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:37:20.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-04-01T05:37:20.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:37:20.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c23e5ba9-46bc-49d8-ac0e-231930823784 2024-04-01T05:37:20.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c23e5ba9-46bc-49d8-ac0e-231930823784' 2024-04-01T05:37:20.581 INFO:tasks.workunit.client.0.smithi067.stdout:add osd4 c23e5ba9-46bc-49d8-ac0e-231930823784 2024-04-01T05:37:20.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:37:20.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQSApm/ih+IxAAtb76g3PqXMZYLyuGQ5v7dQ== 2024-04-01T05:37:20.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQSApm/ih+IxAAtb76g3PqXMZYLyuGQ5v7dQ=="}' 2024-04-01T05:37:20.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c23e5ba9-46bc-49d8-ac0e-231930823784 -i td/osd-scrub-repair/4/new.json 2024-04-01T05:37:20.981 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:37:20.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-04-01T05:37:20.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQSApm/ih+IxAAtb76g3PqXMZYLyuGQ5v7dQ== --osd-uuid c23e5ba9-46bc-49d8-ac0e-231930823784 2024-04-01T05:37:21.021 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.021+0000 7f264d721640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:21.074 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.074+0000 7f264d721640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:21.075 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.075+0000 7f264d721640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:21.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.075+0000 7f264d721640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-04-01T05:37:21.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.075+0000 7f264d721640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-04-01T05:37:21.076 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.075+0000 7f264d721640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-04-01T05:37:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:21.157+0000 7f264d721640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-04-01T05:37:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-04-01T05:37:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:37:27.074 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd4 key to auth repository 2024-04-01T05:37:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-01T05:37:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:37:27.441 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.4 2024-04-01T05:37:27.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-01T05:37:27.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:37:27.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:37:27.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:37:27.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:37:27.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:27.467+0000 7f22f1817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:27.469 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:27.469+0000 7f22f1817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:27.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:27.470+0000 7f22f1817640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-01T05:37:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:37:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-01T05:37:27.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:37:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:37:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:37:27.738 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:37:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:27.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:27.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:27.987+0000 7f22f1817640 -1 Falling back to public interface 2024-04-01T05:37:28.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:29.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:29.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:29.022 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:37:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:37:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:29.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:29.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:30.302 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:30.301+0000 7f22f1817640 -1 osd.4 0 log_to_monitors true 2024-04-01T05:37:30.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:30.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:30.316 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:37:30.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:37:30.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:30.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:30.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:31.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:31.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:31.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:37:31.787 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:37:31.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:31.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:32.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:33.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:33.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:33.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:37:33.194 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:37:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:33.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:33.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:34.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:34.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:34.488 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:37:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:37:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:34.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:37:35.783 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:37:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:36.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:37.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:37.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:37.196 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:37:37.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:37:37.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:37.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:37.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:38.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:38.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:38.490 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:37:38.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:37:38.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:38.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:38.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:39.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:39.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:39.788 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:37:39.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:37:39.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:39.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:40.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:41.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:41.106 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:37:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:37:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:41.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:41.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:42.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:42.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:42.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:37:42.401 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:37:42.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:42.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:42.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:43.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:37:43.695 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:37:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:43.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:44.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:45.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:45.514 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:37:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:37:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:45.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:45.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:46.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:46.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:46.893 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:37:46.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:37:46.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:46.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:47.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:48.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:48.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:48.621 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:37:48.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:37:48.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:48.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:48.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:49.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:49.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:49.981 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:37:49.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:37:49.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:49.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:50.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:51.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:51.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:51.596 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:37:51.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:37:51.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:51.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:51.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:37:52.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:37:52.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:37:52.892 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:37:52.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:37:52.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:37:52.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-01T05:37:53.307 INFO:tasks.workunit.client.0.smithi067.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/228812252,v1:127.0.0.1:6835/228812252] [v2:127.0.0.1:6836/228812252,v1:127.0.0.1:6837/228812252] exists,up c23e5ba9-46bc-49d8-ac0e-231930823784 2024-04-01T05:37:53.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:37:53.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:37:53.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:37:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:37:53.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 5306857f-96f9-4e56-bb5e-552f11be76f2' 2024-04-01T05:37:53.315 INFO:tasks.workunit.client.0.smithi067.stdout:add osd5 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:37:53.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:37:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxSApmIcGVExAAzxV4+1VVbXE8d6ZeO6CcZg== 2024-04-01T05:37:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxSApmIcGVExAAzxV4+1VVbXE8d6ZeO6CcZg=="}' 2024-04-01T05:37:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5306857f-96f9-4e56-bb5e-552f11be76f2 -i td/osd-scrub-repair/5/new.json 2024-04-01T05:37:53.790 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:37:53.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-04-01T05:37:53.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxSApmIcGVExAAzxV4+1VVbXE8d6ZeO6CcZg== --osd-uuid 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:37:53.830 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.831+0000 7ff50a611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:53.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.884+0000 7ff50a611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:53.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.885+0000 7ff50a611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:37:53.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.886+0000 7ff50a611640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-04-01T05:37:53.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.886+0000 7ff50a611640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-04-01T05:37:53.885 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.886+0000 7ff50a611640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-04-01T05:37:53.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:37:53.993+0000 7ff50a611640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-04-01T05:38:00.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-04-01T05:38:00.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:38:00.028 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd5 key to auth repository 2024-04-01T05:38:00.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-01T05:38:00.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:38:00.398 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.5 2024-04-01T05:38:00.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-01T05:38:00.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:38:00.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:38:00.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:38:00.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:38:00.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:00.424+0000 7fd8e64d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:00.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:00.426+0000 7fd8e64d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:00.427 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:00.427+0000 7fd8e64d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-01T05:38:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:38:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:38:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:38:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:38:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:00.682 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:38:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:38:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:00.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:00.942 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:00.943+0000 7fd8e64d5640 -1 Falling back to public interface 2024-04-01T05:38:00.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:38:01.964 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:38:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:01.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:02.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:03.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:03.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:03.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:38:03.257 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:38:03.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:03.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:03.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:03.278+0000 7fd8e64d5640 -1 osd.5 0 log_to_monitors true 2024-04-01T05:38:03.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:04.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:04.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:04.596 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:38:04.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:38:04.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:04.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:04.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:05.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:05.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:05.969 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:38:05.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:38:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:06.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:07.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:07.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:07.264 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:38:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:38:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:07.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:07.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:08.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:08.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:08.557 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:38:08.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:38:08.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:08.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:08.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:09.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:09.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:09.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:38:09.853 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:38:09.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:09.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:10.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:11.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:11.150 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:38:11.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:11.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:38:11.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:11.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:11.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:12.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:12.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:12.451 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:38:12.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:38:12.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:12.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:12.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:13.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:13.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:13.815 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:38:13.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:38:13.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:13.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:14.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:15.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:15.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:15.181 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:38:15.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:38:15.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:15.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:15.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:38:16.735 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:38:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:17.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:18.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:18.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:18.152 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:38:18.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:38:18.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:18.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:19.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:20.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:20.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:20.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:38:20.047 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:38:20.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:20.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:20.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:21.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:21.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:21.738 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:38:21.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:38:21.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:21.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:22.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:23.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:23.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:23.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:38:23.272 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:38:23.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:23.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:23.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:24.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:24.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:24.599 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:38:24.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:38:24.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:24.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:25.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:26.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:26.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:26.005 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:38:26.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:38:26.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:26.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:38:26.295 INFO:tasks.workunit.client.0.smithi067.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3088999246,v1:127.0.0.1:6843/3088999246] [v2:127.0.0.1:6844/3088999246,v1:127.0.0.1:6845/3088999246] exists,up 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:38:26.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:38:26.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:38:26.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f38e8819-3ae7-4414-b1b4-7f2e52fb5313 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 f38e8819-3ae7-4414-b1b4-7f2e52fb5313' 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stdout:add osd6 f38e8819-3ae7-4414-b1b4-7f2e52fb5313 2024-04-01T05:38:26.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:38:26.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSSApmsTXoEhAA2b9kuCJuyWU8smGcVsrYYw== 2024-04-01T05:38:26.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSSApmsTXoEhAA2b9kuCJuyWU8smGcVsrYYw=="}' 2024-04-01T05:38:26.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f38e8819-3ae7-4414-b1b4-7f2e52fb5313 -i td/osd-scrub-repair/6/new.json 2024-04-01T05:38:26.730 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:38:26.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-04-01T05:38:26.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSSApmsTXoEhAA2b9kuCJuyWU8smGcVsrYYw== --osd-uuid f38e8819-3ae7-4414-b1b4-7f2e52fb5313 2024-04-01T05:38:26.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.771+0000 7f5223d14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:26.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.815+0000 7f5223d14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.816+0000 7f5223d14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.817+0000 7f5223d14640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-04-01T05:38:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.817+0000 7f5223d14640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-04-01T05:38:26.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.817+0000 7f5223d14640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-04-01T05:38:26.886 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:26.887+0000 7f5223d14640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-04-01T05:38:33.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-04-01T05:38:33.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:38:33.063 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd6 key to auth repository 2024-04-01T05:38:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-04-01T05:38:33.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:38:33.437 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.6 2024-04-01T05:38:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-04-01T05:38:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:38:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:38:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:38:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:38:33.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:33.464+0000 7f7625738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:33.465 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:33.466+0000 7f7625738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:33.466 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:33.467+0000 7f7625738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:33.755 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:38:33.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:38:33.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:33.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:33.979 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:33.980+0000 7f7625738640 -1 Falling back to public interface 2024-04-01T05:38:34.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:35.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:35.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:35.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:38:35.037 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:38:35.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:35.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:35.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:36.319 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:38:36.319+0000 7f7625738640 -1 osd.6 0 log_to_monitors true 2024-04-01T05:38:36.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:36.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:36.331 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:38:36.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:38:36.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:36.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:36.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:37.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:37.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:37.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:38:37.628 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:38:37.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:37.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:38.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:39.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:39.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:39.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:38:39.005 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:38:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:39.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:39.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:40.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:40.379 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:38:40.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:40.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:38:40.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:40.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:40.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:41.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:41.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:41.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:38:41.673 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:38:41.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:41.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:42.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:43.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:43.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:43.038 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:38:43.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:38:43.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:43.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:43.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:44.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:44.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:44.446 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:38:44.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:38:44.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:44.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:44.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:45.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:45.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:45.685 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:38:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:38:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:46.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:47.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:47.223 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:38:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:38:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:47.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:47.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:48.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:48.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:48.519 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:38:48.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:38:48.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:48.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:49.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:50.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:50.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:50.033 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:38:50.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:38:50.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:50.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:50.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:51.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:51.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:51.604 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:38:51.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:38:51.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:51.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:52.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:53.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:53.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:53.217 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:38:53.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:38:53.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:53.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:53.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:54.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:54.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:54.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:38:54.937 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:38:54.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:54.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:55.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:56.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:56.330 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:38:56.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:56.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:38:56.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:56.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:56.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:57.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:57.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:57.950 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:38:57.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:38:57.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:57.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:58.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:38:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:38:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:38:59.675 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:38:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:38:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:38:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-04-01T05:38:59.993 INFO:tasks.workunit.client.0.smithi067.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3977455255,v1:127.0.0.1:6851/3977455255] [v2:127.0.0.1:6852/3977455255,v1:127.0.0.1:6853/3977455255] exists,up f38e8819-3ae7-4414-b1b4-7f2e52fb5313 2024-04-01T05:38:59.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:38:59.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-04-01T05:38:59.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:38:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:39:00.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc8ebc27-89d4-42e1-8f95-6600807b01e9 2024-04-01T05:39:00.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 fc8ebc27-89d4-42e1-8f95-6600807b01e9' 2024-04-01T05:39:00.001 INFO:tasks.workunit.client.0.smithi067.stdout:add osd7 fc8ebc27-89d4-42e1-8f95-6600807b01e9 2024-04-01T05:39:00.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:39:00.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0SApmOhjuABAAoezyhee0Y6ZMoTzpOBldUQ== 2024-04-01T05:39:00.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0SApmOhjuABAAoezyhee0Y6ZMoTzpOBldUQ=="}' 2024-04-01T05:39:00.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc8ebc27-89d4-42e1-8f95-6600807b01e9 -i td/osd-scrub-repair/7/new.json 2024-04-01T05:39:00.347 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:39:00.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-04-01T05:39:00.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0SApmOhjuABAAoezyhee0Y6ZMoTzpOBldUQ== --osd-uuid fc8ebc27-89d4-42e1-8f95-6600807b01e9 2024-04-01T05:39:00.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.387+0000 7f0ab0f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:00.448 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.449+0000 7f0ab0f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.450+0000 7f0ab0f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.450+0000 7f0ab0f45640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-04-01T05:39:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.450+0000 7f0ab0f45640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-04-01T05:39:00.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.450+0000 7f0ab0f45640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-04-01T05:39:00.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:00.527+0000 7f0ab0f45640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-04-01T05:39:07.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-04-01T05:39:07.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:39:07.017 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd7 key to auth repository 2024-04-01T05:39:07.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-04-01T05:39:07.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:39:07.390 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.7 2024-04-01T05:39:07.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-04-01T05:39:07.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:39:07.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:39:07.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:39:07.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:39:07.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:07.416+0000 7f52ec747640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:07.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:07.418+0000 7f52ec747640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:07.419 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:07.420+0000 7f52ec747640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:07.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-04-01T05:39:07.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:39:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:07.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:07.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:07.936+0000 7f52ec747640 -1 Falling back to public interface 2024-04-01T05:39:07.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:08.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:08.965 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:39:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:39:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:08.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:09.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:10.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:10.258 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:39:10.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:10.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:39:10.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:10.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:10.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:10.281+0000 7f52ec747640 -1 osd.7 0 log_to_monitors true 2024-04-01T05:39:10.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:11.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:11.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:11.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:39:11.605 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:39:11.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:11.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:11.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:12.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:12.930 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:39:12.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:12.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:39:12.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:12.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:13.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:14.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:14.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:14.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:39:14.219 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:39:14.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:14.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:14.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:15.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:15.523 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:39:15.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:15.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:39:15.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:15.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:15.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:16.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:16.899 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:39:16.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:39:16.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:16.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:17.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:18.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:18.193 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:39:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:39:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:18.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:19.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:19.535 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:39:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:39:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:19.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:20.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:20.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:20.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:39:20.901 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:39:20.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:20.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:21.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:22.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:22.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:22.199 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:39:22.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:39:22.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:22.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:22.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:23.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:23.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:23.493 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:39:23.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:39:23.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:23.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:24.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:25.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:25.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:25.131 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:39:25.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:39:25.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:25.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:25.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:26.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:26.818 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:39:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:39:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:26.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:27.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:28.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:28.588 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:39:28.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:28.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:39:28.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:28.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:29.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:30.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:30.045 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:39:30.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:30.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:39:30.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:30.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:30.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:31.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:31.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:31.616 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:39:31.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:39:31.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:31.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:32.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:33.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:33.182 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:39:33.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:33.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:39:33.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:33.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:33.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:34.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:34.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:34.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:39:34.495 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:39:34.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:34.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-04-01T05:39:34.788 INFO:tasks.workunit.client.0.smithi067.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4206957564,v1:127.0.0.1:6859/4206957564] [v2:127.0.0.1:6860/4206957564,v1:127.0.0.1:6861/4206957564] exists,up fc8ebc27-89d4-42e1-8f95-6600807b01e9 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:39:34.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-04-01T05:39:34.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:39:34.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:39:34.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:39:34.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:39:34.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:39:34.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c29bc3a3-8ffd-4055-bfe4-e1999c662b58 2024-04-01T05:39:34.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 c29bc3a3-8ffd-4055-bfe4-e1999c662b58' 2024-04-01T05:39:34.797 INFO:tasks.workunit.client.0.smithi067.stdout:add osd8 c29bc3a3-8ffd-4055-bfe4-e1999c662b58 2024-04-01T05:39:34.797 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:39:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWSApm1c5MMBAAXSitCFokKpwl5mK43gt/EA== 2024-04-01T05:39:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWSApm1c5MMBAAXSitCFokKpwl5mK43gt/EA=="}' 2024-04-01T05:39:34.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c29bc3a3-8ffd-4055-bfe4-e1999c662b58 -i td/osd-scrub-repair/8/new.json 2024-04-01T05:39:35.153 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:39:35.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-04-01T05:39:35.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWSApm1c5MMBAAXSitCFokKpwl5mK43gt/EA== --osd-uuid c29bc3a3-8ffd-4055-bfe4-e1999c662b58 2024-04-01T05:39:35.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.193+0000 7ff9d891d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:35.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.204+0000 7ff9d891d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:35.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.205+0000 7ff9d891d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:35.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.205+0000 7ff9d891d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-04-01T05:39:35.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.205+0000 7ff9d891d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-04-01T05:39:35.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.205+0000 7ff9d891d640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-04-01T05:39:35.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:35.304+0000 7ff9d891d640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-04-01T05:39:41.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-04-01T05:39:41.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:39:41.542 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd8 key to auth repository 2024-04-01T05:39:41.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-04-01T05:39:41.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:39:41.933 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.8 2024-04-01T05:39:41.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-04-01T05:39:41.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:39:41.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:39:41.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:39:41.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:39:41.959 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:41.959+0000 7f3fa4e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:41.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:41.961+0000 7f3fa4e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:41.962 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:41.962+0000 7f3fa4e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:39:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-04-01T05:39:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:39:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-04-01T05:39:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:39:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:39:42.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:42.222 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:39:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:39:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:42.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:42.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:42.476+0000 7f3fa4e0c640 -1 Falling back to public interface 2024-04-01T05:39:42.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:43.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:43.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:43.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:39:43.505 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:39:43.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:43.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:43.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:44.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:44.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:44.799 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:39:44.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:39:44.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:44.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:44.877 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:39:44.878+0000 7f3fa4e0c640 -1 osd.8 0 log_to_monitors true 2024-04-01T05:39:45.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:46.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:46.154 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:39:46.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:46.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:39:46.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:46.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:46.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:39:47.451 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:39:47.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:47.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:47.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:48.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:48.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:48.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:39:48.792 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:39:48.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:48.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:49.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:50.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:50.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:50.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:39:50.086 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:39:50.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:50.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:50.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:51.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:51.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:51.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:39:51.382 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:39:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:51.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:52.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:52.756 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:39:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:39:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:52.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:53.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:54.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:54.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:54.112 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:39:54.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:39:54.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:54.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:54.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:39:55.405 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:39:55.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:55.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:55.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:56.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:56.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:56.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:39:56.750 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:39:56.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:56.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:57.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:58.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:58.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:58.252 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:39:58.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:39:58.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:58.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:39:58.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:39:59.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:39:59.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:39:59.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:39:59.924 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:39:59.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:39:59.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:40:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:01.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:01.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:01.535 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:40:01.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:40:01.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:01.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:40:01.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:02.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:02.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:02.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:40:02.911 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:40:02.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:02.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:40:03.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:04.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:04.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:04.658 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:40:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:40:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:04.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:40:05.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:06.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:06.153 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:40:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:40:06.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:06.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:40:06.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:07.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:07.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:07.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:40:07.858 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:40:07.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:07.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-04-01T05:40:08.189 INFO:tasks.workunit.client.0.smithi067.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/598193704,v1:127.0.0.1:6867/598193704] [v2:127.0.0.1:6868/598193704,v1:127.0.0.1:6869/598193704] exists,up c29bc3a3-8ffd-4055-bfe4-e1999c662b58 2024-04-01T05:40:08.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:40:08.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:40:08.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:40:08.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:40:08.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:40:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:40:08.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 0726d7fc-af3e-41b8-b7d7-d599ec6371e9' 2024-04-01T05:40:08.197 INFO:tasks.workunit.client.0.smithi067.stdout:add osd9 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:40:08.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:40:08.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4SApmCPaNDBAA7hlTcM0tbQPsflKykBrVhA== 2024-04-01T05:40:08.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4SApmCPaNDBAA7hlTcM0tbQPsflKykBrVhA=="}' 2024-04-01T05:40:08.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 -i td/osd-scrub-repair/9/new.json 2024-04-01T05:40:08.671 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:40:08.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-04-01T05:40:08.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4SApmCPaNDBAA7hlTcM0tbQPsflKykBrVhA== --osd-uuid 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:40:08.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.712+0000 7f0ed6723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:40:08.773 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.773+0000 7f0ed6723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:40:08.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.775+0000 7f0ed6723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:40:08.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.775+0000 7f0ed6723640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-04-01T05:40:08.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.775+0000 7f0ed6723640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-04-01T05:40:08.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.775+0000 7f0ed6723640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-04-01T05:40:08.898 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:08.899+0000 7f0ed6723640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-04-01T05:40:15.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-04-01T05:40:15.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:40:15.210 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd9 key to auth repository 2024-04-01T05:40:15.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-04-01T05:40:15.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:40:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-04-01T05:40:15.578 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.9 2024-04-01T05:40:15.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:40:15.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:40:15.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:40:15.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:40:15.603 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:15.604+0000 7f8a2ae13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:40:15.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:15.606+0000 7f8a2ae13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:40:15.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:15.607+0000 7f8a2ae13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:40:15.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-04-01T05:40:15.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:40:15.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:15.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:16.121 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:16.122+0000 7f8a2ae13640 -1 Falling back to public interface 2024-04-01T05:40:16.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:17.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:17.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:17.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:40:17.146 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:40:17.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:17.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:17.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:18.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:18.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:18.437 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:40:18.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:40:18.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:18.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:18.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:40:18.462+0000 7f8a2ae13640 -1 osd.9 0 log_to_monitors true 2024-04-01T05:40:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:19.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:19.781 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:40:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:40:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:20.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:21.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:21.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:21.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:40:21.125 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:40:21.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:21.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:21.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:22.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:22.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:22.468 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:40:22.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:40:22.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:22.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:22.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:23.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:23.763 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:40:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:40:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:23.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:24.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:25.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:25.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:25.126 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:40:25.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:40:25.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:25.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:25.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:26.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:26.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:26.431 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:40:26.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:40:26.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:26.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:26.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:27.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:27.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:27.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:40:27.724 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:40:27.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:27.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:28.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:29.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:29.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:29.091 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:40:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:40:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:29.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:30.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:30.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:30.451 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:40:30.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:40:30.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:30.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:30.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:31.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:31.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:31.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:40:31.745 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:40:31.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:31.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:32.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:33.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:33.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:33.080 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:40:33.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:40:33.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:33.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:33.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:34.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:34.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:34.440 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:40:34.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:40:34.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:34.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:35.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:36.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:36.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:36.060 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:40:36.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:40:36.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:36.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:36.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:37.573 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:40:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:40:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:38.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:39.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:39.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:39.084 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:40:39.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:40:39.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:39.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:39.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:40.460 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:40:40.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:40:40.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:40.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:40.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:40:41.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:40:41.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:40:41.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:40:41.868 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:40:41.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:40:41.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:40:42.165 INFO:tasks.workunit.client.0.smithi067.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1099327942,v1:127.0.0.1:6875/1099327942] [v2:127.0.0.1:6876/1099327942,v1:127.0.0.1:6877/1099327942] exists,up 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:40:42.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:40:42.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:40:42.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:40:42.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-04-01T05:40:42.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:40:42.501 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:40:42.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:40:42.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:40:42.979 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:40:42.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:40:43.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:40:47.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-04-01T05:40:47.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:40:47.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:40:47.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:40:47.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:40:47.203 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:40:47.203 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:40:47.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:40:47.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:40:47.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:40:47.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:40:47.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:40:47.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:40:47.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:40:47.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:40:47.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:40:47.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:40:47.583 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:40:47.583 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:47.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:40:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-04-01T05:40:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-04-01T05:40:47.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-04-01T05:40:47.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:47.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:40:47.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-04-01T05:40:47.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-04-01T05:40:47.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020' 2024-04-01T05:40:47.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:47.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:40:47.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509493 2024-04-01T05:40:47.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509493 2024-04-01T05:40:47.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493' 2024-04-01T05:40:47.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:47.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:40:48.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345965 2024-04-01T05:40:48.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345965 2024-04-01T05:40:48.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965' 2024-04-01T05:40:48.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:48.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:40:48.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182441 2024-04-01T05:40:48.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182441 2024-04-01T05:40:48.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965 4-107374182441' 2024-04-01T05:40:48.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:48.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:40:48.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018911 2024-04-01T05:40:48.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018911 2024-04-01T05:40:48.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965 4-107374182441 5-128849018911' 2024-04-01T05:40:48.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:48.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:40:48.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855385 2024-04-01T05:40:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855385 2024-04-01T05:40:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965 4-107374182441 5-128849018911 6-150323855385' 2024-04-01T05:40:48.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:48.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:40:48.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-04-01T05:40:48.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-04-01T05:40:48.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965 4-107374182441 5-128849018911 6-150323855385 7-171798691861' 2024-04-01T05:40:48.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:48.430 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:40:48.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528334 2024-04-01T05:40:48.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528334 2024-04-01T05:40:48.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965 4-107374182441 5-128849018911 6-150323855385 7-171798691861 8-193273528334' 2024-04-01T05:40:48.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:48.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:40:48.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364805 2024-04-01T05:40:48.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364805 2024-04-01T05:40:48.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673020 2-64424509493 3-85899345965 4-107374182441 5-128849018911 6-150323855385 7-171798691861 8-193273528334 9-214748364805' 2024-04-01T05:40:48.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:48.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-04-01T05:40:48.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:48.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:40:48.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-04-01T05:40:48.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:48.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-04-01T05:40:48.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-04-01T05:40:48.642 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836550 2024-04-01T05:40:48.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:40:48.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-04-01T05:40:48.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:48.925 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-04-01T05:40:48.925 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:48.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:40:48.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-04-01T05:40:48.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:48.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-04-01T05:40:48.928 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949673020 2024-04-01T05:40:48.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-04-01T05:40:48.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:40:49.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673020 2024-04-01T05:40:49.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:49.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509493 2024-04-01T05:40:49.213 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:49.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:40:49.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509493 2024-04-01T05:40:49.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:49.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509493 2024-04-01T05:40:49.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-04-01T05:40:49.217 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509493 2024-04-01T05:40:49.217 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:40:49.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509493 -lt 64424509493 2024-04-01T05:40:49.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:49.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345965 2024-04-01T05:40:49.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:49.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:40:49.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345965 2024-04-01T05:40:49.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:49.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345965 2024-04-01T05:40:49.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2024-04-01T05:40:49.502 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345965 2024-04-01T05:40:49.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:40:49.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345965 -lt 85899345965 2024-04-01T05:40:49.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:49.829 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182441 2024-04-01T05:40:49.829 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:49.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:40:49.830 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182441 2024-04-01T05:40:49.830 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:49.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182441 2024-04-01T05:40:49.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2024-04-01T05:40:49.831 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.4 seq 107374182441 2024-04-01T05:40:49.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:40:50.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182441 -lt 107374182441 2024-04-01T05:40:50.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:50.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018911 2024-04-01T05:40:50.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:50.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:40:50.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018911 2024-04-01T05:40:50.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:50.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018911 2024-04-01T05:40:50.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018911' 2024-04-01T05:40:50.142 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.5 seq 128849018911 2024-04-01T05:40:50.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:40:50.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018911 -lt 128849018911 2024-04-01T05:40:50.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:50.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855385 2024-04-01T05:40:50.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:50.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:40:50.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855385 2024-04-01T05:40:50.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:50.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855385 2024-04-01T05:40:50.428 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.6 seq 150323855385 2024-04-01T05:40:50.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855385' 2024-04-01T05:40:50.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:40:50.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855385 -lt 150323855385 2024-04-01T05:40:50.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-04-01T05:40:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:50.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:40:50.713 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-04-01T05:40:50.713 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:50.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-04-01T05:40:50.714 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.7 seq 171798691861 2024-04-01T05:40:50.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-04-01T05:40:50.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:40:51.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691861 2024-04-01T05:40:51.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:51.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528334 2024-04-01T05:40:51.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:51.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:40:51.003 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528334 2024-04-01T05:40:51.003 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:51.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528334 2024-04-01T05:40:51.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528334' 2024-04-01T05:40:51.004 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.8 seq 193273528334 2024-04-01T05:40:51.004 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:40:51.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528334 -lt 193273528334 2024-04-01T05:40:51.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:51.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364805 2024-04-01T05:40:51.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:51.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:40:51.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364805 2024-04-01T05:40:51.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:51.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364805 2024-04-01T05:40:51.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364805' 2024-04-01T05:40:51.307 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.9 seq 214748364805 2024-04-01T05:40:51.307 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:40:51.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364805 -lt 214748364805 2024-04-01T05:40:51.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:40:51.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:40:51.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:40:52.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:40:52.008 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:40:52.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:40:52.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:40:52.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:40:52.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-04-01T05:40:52.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:40:52.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-04-01T05:40:53.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:40:53.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:40:53.804 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:40:53.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:40:54.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-04-01T05:40:54.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-04-01T05:40:55.750 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 2 allow_ec_overwrites to true 2024-04-01T05:40:55.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:40:55.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:40:55.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:40:55.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:40:55.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:40:55.762 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:40:55.762 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:40:55.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:40:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:40:55.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:40:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:40:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:40:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:40:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:40:55.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:40:55.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:40:56.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:56.442 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:40:56.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-04-01T05:40:56.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-04-01T05:40:56.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-04-01T05:40:56.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:56.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:40:56.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673023 2024-04-01T05:40:56.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673023 2024-04-01T05:40:56.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023' 2024-04-01T05:40:56.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:56.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:40:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-04-01T05:40:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-04-01T05:40:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496' 2024-04-01T05:40:56.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:56.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:40:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345968 2024-04-01T05:40:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345968 2024-04-01T05:40:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968' 2024-04-01T05:40:57.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:57.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:40:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182444 2024-04-01T05:40:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182444 2024-04-01T05:40:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968 4-107374182444' 2024-04-01T05:40:57.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:57.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:40:57.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-04-01T05:40:57.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-04-01T05:40:57.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968 4-107374182444 5-128849018914' 2024-04-01T05:40:57.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:57.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:40:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855388 2024-04-01T05:40:57.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855388 2024-04-01T05:40:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968 4-107374182444 5-128849018914 6-150323855388' 2024-04-01T05:40:57.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:57.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:40:57.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-04-01T05:40:57.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-04-01T05:40:57.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968 4-107374182444 5-128849018914 6-150323855388 7-171798691864' 2024-04-01T05:40:57.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:57.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:40:57.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528336 2024-04-01T05:40:57.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528336 2024-04-01T05:40:57.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968 4-107374182444 5-128849018914 6-150323855388 7-171798691864 8-193273528336' 2024-04-01T05:40:57.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:40:57.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:40:57.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364808 2024-04-01T05:40:57.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364808 2024-04-01T05:40:57.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673023 2-64424509496 3-85899345968 4-107374182444 5-128849018914 6-150323855388 7-171798691864 8-193273528336 9-214748364808' 2024-04-01T05:40:57.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:57.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-04-01T05:40:57.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:57.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:40:57.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-04-01T05:40:57.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:57.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-04-01T05:40:57.659 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836553 2024-04-01T05:40:57.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-04-01T05:40:57.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:40:57.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-04-01T05:40:57.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:57.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673023 2024-04-01T05:40:57.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:57.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:40:57.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673023 2024-04-01T05:40:57.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:57.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673023 2024-04-01T05:40:57.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-04-01T05:40:57.945 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949673023 2024-04-01T05:40:57.945 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:40:58.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673022 -lt 42949673023 2024-04-01T05:40:58.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:40:59.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:40:59.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:40:59.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673024 -lt 42949673023 2024-04-01T05:40:59.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:59.527 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-04-01T05:40:59.527 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:59.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:40:59.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-04-01T05:40:59.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:59.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-04-01T05:40:59.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-04-01T05:40:59.530 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509496 2024-04-01T05:40:59.530 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:40:59.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-04-01T05:40:59.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:40:59.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345968 2024-04-01T05:40:59.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:40:59.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:40:59.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345968 2024-04-01T05:40:59.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:40:59.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345968 2024-04-01T05:40:59.823 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.3 seq 85899345968 2024-04-01T05:40:59.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2024-04-01T05:40:59.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:41:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345968 2024-04-01T05:41:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:00.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182444 2024-04-01T05:41:00.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:41:00.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182444 2024-04-01T05:41:00.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:00.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182444 2024-04-01T05:41:00.111 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.4 seq 107374182444 2024-04-01T05:41:00.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2024-04-01T05:41:00.111 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:41:00.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182444 -lt 107374182444 2024-04-01T05:41:00.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:00.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018914 2024-04-01T05:41:00.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:00.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:41:00.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018914 2024-04-01T05:41:00.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:00.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-04-01T05:41:00.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018914' 2024-04-01T05:41:00.401 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.5 seq 128849018914 2024-04-01T05:41:00.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:41:00.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018914 -lt 128849018914 2024-04-01T05:41:00.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:00.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855388 2024-04-01T05:41:00.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:00.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:41:00.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855388 2024-04-01T05:41:00.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:00.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855388 2024-04-01T05:41:00.689 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.6 seq 150323855388 2024-04-01T05:41:00.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-04-01T05:41:00.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:41:00.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855388 2024-04-01T05:41:00.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:00.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691864 2024-04-01T05:41:00.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:00.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:41:00.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691864 2024-04-01T05:41:00.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:00.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-04-01T05:41:00.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-04-01T05:41:00.981 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.7 seq 171798691864 2024-04-01T05:41:00.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:41:01.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-04-01T05:41:01.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:01.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528336 2024-04-01T05:41:01.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:01.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:41:01.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528336 2024-04-01T05:41:01.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:01.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528336 2024-04-01T05:41:01.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-04-01T05:41:01.272 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.8 seq 193273528336 2024-04-01T05:41:01.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:41:01.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528337 -lt 193273528336 2024-04-01T05:41:01.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:01.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364808 2024-04-01T05:41:01.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:01.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:41:01.557 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364808 2024-04-01T05:41:01.557 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364808 2024-04-01T05:41:01.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-04-01T05:41:01.558 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.9 seq 214748364808 2024-04-01T05:41:01.558 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:41:01.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364808 -lt 214748364808 2024-04-01T05:41:01.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:41:01.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:01.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:02.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:41:02.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:41:02.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:41:02.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:41:02.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:02.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T05:41:02.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:41:02.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:41:03.226 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:41:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:41:03.746 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:41:03.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:41:03.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:41:03.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:41:04.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-04-01T05:41:04.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-04-01T05:41:04.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:41:04.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:41:04.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:41:04.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-04-01T05:41:04.477 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-04-01T05:41:04.477 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T05:41:04.477 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-04-01T05:41:04.477 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-04-01T05:41:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:41:04.478 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:9 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:7' 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-04-01T05:41:04.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:41:04.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:41:04.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:41:05.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:41:05.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:41:06.976 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:41:07.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:41:07.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:41:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:41:07.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.3 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:41:07.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:41:07.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:07.526+0000 7f47c581c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:07.528 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:07.528+0000 7f47c581c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:07.530 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:07.529+0000 7f47c581c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:41:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:41:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:41:07.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:41:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:41:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:41:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:41:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:07.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:41:08.055 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:08.055+0000 7f47c581c640 -1 Falling back to public interface 2024-04-01T05:41:08.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:09.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:41:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:09.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:41:09.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:10.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:10.401+0000 7f47c581c640 -1 osd.3 68 log_to_monitors true 2024-04-01T05:41:10.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:10.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:10.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:41:10.407 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:10.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:10.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:41:10.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:11.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:41:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:41:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:11.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/4091429379,v1:127.0.0.1:6827/4091429379] [v2:127.0.0.1:6828/4091429379,v1:127.0.0.1:6829/4091429379] exists,up f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:41:12.037 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:41:12.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:41:12.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:41:12.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:41:12.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:41:12.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:41:12.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:41:12.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:41:12.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:41:12.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:41:12.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:12.411 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:41:12.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-04-01T05:41:12.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-04-01T05:41:12.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-04-01T05:41:12.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:12.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:41:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-04-01T05:41:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-04-01T05:41:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027' 2024-04-01T05:41:12.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:12.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:41:12.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509500 2024-04-01T05:41:12.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509500 2024-04-01T05:41:12.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500' 2024-04-01T05:41:12.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:12.924 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:41:13.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-04-01T05:41:13.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-04-01T05:41:13.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314' 2024-04-01T05:41:13.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:13.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:41:13.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-04-01T05:41:13.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-04-01T05:41:13.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314 4-107374182448' 2024-04-01T05:41:13.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:13.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:41:13.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-04-01T05:41:13.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-04-01T05:41:13.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314 4-107374182448 5-128849018918' 2024-04-01T05:41:13.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:13.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:41:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855392 2024-04-01T05:41:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855392 2024-04-01T05:41:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314 4-107374182448 5-128849018918 6-150323855392' 2024-04-01T05:41:13.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:13.346 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:41:13.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-04-01T05:41:13.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-04-01T05:41:13.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314 4-107374182448 5-128849018918 6-150323855392 7-171798691868' 2024-04-01T05:41:13.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:13.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:41:13.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528341 2024-04-01T05:41:13.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528341 2024-04-01T05:41:13.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314 4-107374182448 5-128849018918 6-150323855392 7-171798691868 8-193273528341' 2024-04-01T05:41:13.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:13.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:41:13.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-04-01T05:41:13.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-04-01T05:41:13.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673027 2-64424509500 3-309237645314 4-107374182448 5-128849018918 6-150323855392 7-171798691868 8-193273528341 9-214748364812' 2024-04-01T05:41:13.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:13.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-04-01T05:41:13.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:13.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:41:13.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-04-01T05:41:13.663 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:13.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-04-01T05:41:13.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-04-01T05:41:13.664 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836557 2024-04-01T05:41:13.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:41:13.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-04-01T05:41:13.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:13.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-04-01T05:41:13.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:13.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:41:13.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-04-01T05:41:13.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:13.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-04-01T05:41:13.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-04-01T05:41:13.959 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673027 2024-04-01T05:41:13.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:41:14.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673027 2024-04-01T05:41:14.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:41:15.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:41:15.244 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:41:15.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673028 -lt 42949673027 2024-04-01T05:41:15.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:15.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509500 2024-04-01T05:41:15.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:15.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:41:15.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509500 2024-04-01T05:41:15.541 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:15.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509500 2024-04-01T05:41:15.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-04-01T05:41:15.542 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509500 2024-04-01T05:41:15.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:41:15.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509500 -lt 64424509500 2024-04-01T05:41:15.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:15.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645314 2024-04-01T05:41:15.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:15.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:41:15.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645314 2024-04-01T05:41:15.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:15.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-04-01T05:41:15.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-04-01T05:41:15.845 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 309237645314 2024-04-01T05:41:15.846 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:41:16.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-04-01T05:41:16.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:16.160 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-04-01T05:41:16.161 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:16.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:41:16.162 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-04-01T05:41:16.162 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:16.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-04-01T05:41:16.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-04-01T05:41:16.163 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182448 2024-04-01T05:41:16.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:41:16.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182449 -lt 107374182448 2024-04-01T05:41:16.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:16.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018918 2024-04-01T05:41:16.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:16.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:41:16.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018918 2024-04-01T05:41:16.448 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-04-01T05:41:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-04-01T05:41:16.449 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 128849018918 2024-04-01T05:41:16.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:41:16.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018918 -lt 128849018918 2024-04-01T05:41:16.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:16.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855392 2024-04-01T05:41:16.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:16.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:41:16.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855392 2024-04-01T05:41:16.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:16.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855392 2024-04-01T05:41:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-04-01T05:41:16.735 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855392 2024-04-01T05:41:16.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:41:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855393 -lt 150323855392 2024-04-01T05:41:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:17.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-04-01T05:41:17.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:41:17.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-04-01T05:41:17.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-04-01T05:41:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-04-01T05:41:17.048 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691868 2024-04-01T05:41:17.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:41:17.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-04-01T05:41:17.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:17.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528341 2024-04-01T05:41:17.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:17.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:41:17.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528341 2024-04-01T05:41:17.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:17.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528341 2024-04-01T05:41:17.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-04-01T05:41:17.336 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528341 2024-04-01T05:41:17.336 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:41:17.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528341 -lt 193273528341 2024-04-01T05:41:17.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:17.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-04-01T05:41:17.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:17.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:41:17.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-04-01T05:41:17.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:17.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-04-01T05:41:17.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-04-01T05:41:17.622 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 214748364812 2024-04-01T05:41:17.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:41:17.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364812 2024-04-01T05:41:17.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:41:17.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:17.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:41:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:41:18.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:41:18.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:41:18.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:41:18.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:41:18.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:41:18.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:41:18.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:41:18.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:41:18.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:18.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:41:18.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:41:18.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:19.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:19.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:19.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:41:19.789 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:41:19.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:19.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:20.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:40:53.628593+0000 '>' 2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:20.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:21.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:21.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:21.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:40:53.628593+0000 '>' 2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:21.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:22.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:22.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:22.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:22.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:22.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:22.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:22.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:22.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:40:53.628593+0000 '>' 2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:22.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:23.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:23.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:20.484147+0000 '>' 2024-04-01T05:40:53.628593+0000 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:41:23.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:41:23.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:41:24.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:41:24.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:41:24.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:41:24.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:41:24.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:41:24.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:41:24.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:41:24.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-04-01T05:41:26.174 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:41:26.174 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:41:26.174 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:41:26.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:41:26.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:41:26.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:41:26.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:41:26.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:41:26.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:41:26.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:41:26.446 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.9 2024-04-01T05:41:26.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:41:26.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:41:26.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:41:26.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:41:26.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:41:26.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:41:26.474 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:26.474+0000 7fefba813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:26.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:26.476+0000 7fefba813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:26.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:26.478+0000 7fefba813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:41:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:41:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:41:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:41:26.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:41:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:41:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:41:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:26.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:41:26.999 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:27.000+0000 7fefba813640 -1 Falling back to public interface 2024-04-01T05:41:27.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:28.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:28.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:28.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:41:28.066 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:28.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:28.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:41:28.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:29.314 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:29.315+0000 7fefba813640 -1 osd.9 73 log_to_monitors true 2024-04-01T05:41:29.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:29.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:29.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:41:29.360 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:29.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:29.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:41:29.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:41:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:41:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:41:31.271 INFO:tasks.workunit.client.0.smithi067.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/1472507644,v1:127.0.0.1:6875/1472507644] [v2:127.0.0.1:6876/1472507644,v1:127.0.0.1:6877/1472507644] exists,up 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:41:31.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:41:31.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:41:31.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:41:31.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:41:31.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:41:31.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:41:31.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:41:31.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:41:31.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:41:31.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:41:31.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:31.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:41:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-04-01T05:41:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-04-01T05:41:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-04-01T05:41:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:31.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:41:32.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-04-01T05:41:32.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-04-01T05:41:32.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032' 2024-04-01T05:41:32.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:41:32.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-04-01T05:41:32.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-04-01T05:41:32.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505' 2024-04-01T05:41:32.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:41:32.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645319 2024-04-01T05:41:32.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645319 2024-04-01T05:41:32.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319' 2024-04-01T05:41:32.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.247 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:41:32.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182453 2024-04-01T05:41:32.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182453 2024-04-01T05:41:32.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319 4-107374182453' 2024-04-01T05:41:32.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:41:32.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-04-01T05:41:32.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-04-01T05:41:32.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319 4-107374182453 5-128849018923' 2024-04-01T05:41:32.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:41:32.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-04-01T05:41:32.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-04-01T05:41:32.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319 4-107374182453 5-128849018923 6-150323855397' 2024-04-01T05:41:32.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:41:32.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-04-01T05:41:32.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-04-01T05:41:32.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319 4-107374182453 5-128849018923 6-150323855397 7-171798691873' 2024-04-01T05:41:32.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:41:32.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-04-01T05:41:32.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-04-01T05:41:32.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319 4-107374182453 5-128849018923 6-150323855397 7-171798691873 8-193273528345' 2024-04-01T05:41:32.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:32.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:41:32.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-04-01T05:41:32.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-04-01T05:41:32.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673032 2-64424509505 3-309237645319 4-107374182453 5-128849018923 6-150323855397 7-171798691873 8-193273528345 9-330712481794' 2024-04-01T05:41:32.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:32.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-04-01T05:41:32.911 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:32.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:41:32.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-04-01T05:41:32.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:32.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-04-01T05:41:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-04-01T05:41:32.914 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836562 2024-04-01T05:41:32.914 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:41:33.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836562 2024-04-01T05:41:33.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:33.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-04-01T05:41:33.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:33.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:41:33.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-04-01T05:41:33.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-04-01T05:41:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-04-01T05:41:33.200 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673032 2024-04-01T05:41:33.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-04-01T05:41:33.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:33.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-04-01T05:41:33.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-04-01T05:41:33.488 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509505 2024-04-01T05:41:33.488 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:41:33.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509505 -lt 64424509505 2024-04-01T05:41:33.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:33.854 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645319 2024-04-01T05:41:33.854 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:33.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:41:33.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645319 2024-04-01T05:41:33.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:33.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645319 2024-04-01T05:41:33.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-04-01T05:41:33.857 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 309237645319 2024-04-01T05:41:33.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:41:34.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645319 -lt 309237645319 2024-04-01T05:41:34.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:34.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182453 2024-04-01T05:41:34.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:34.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:41:34.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182453 2024-04-01T05:41:34.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:34.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182453 2024-04-01T05:41:34.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182453' 2024-04-01T05:41:34.143 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182453 2024-04-01T05:41:34.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:41:34.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182453 -lt 107374182453 2024-04-01T05:41:34.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:34.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018923 2024-04-01T05:41:34.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:34.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:41:34.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018923 2024-04-01T05:41:34.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:34.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-04-01T05:41:34.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-04-01T05:41:34.430 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 128849018923 2024-04-01T05:41:34.430 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:41:34.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018923 -lt 128849018923 2024-04-01T05:41:34.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:34.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-04-01T05:41:34.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:34.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:41:34.716 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-04-01T05:41:34.716 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:34.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-04-01T05:41:34.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-04-01T05:41:34.717 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855397 2024-04-01T05:41:34.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:41:35.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855397 2024-04-01T05:41:35.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:35.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-04-01T05:41:35.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:35.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:41:35.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-04-01T05:41:35.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-04-01T05:41:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-04-01T05:41:35.008 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691873 2024-04-01T05:41:35.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:41:35.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691873 -lt 171798691873 2024-04-01T05:41:35.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:35.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-04-01T05:41:35.301 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:35.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:41:35.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-04-01T05:41:35.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:35.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-04-01T05:41:35.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-04-01T05:41:35.303 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528345 2024-04-01T05:41:35.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:41:35.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528345 2024-04-01T05:41:35.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:35.589 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-04-01T05:41:35.589 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:35.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:41:35.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-04-01T05:41:35.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:35.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-04-01T05:41:35.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-04-01T05:41:35.592 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 330712481794 2024-04-01T05:41:35.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:41:35.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-04-01T05:41:35.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:41:35.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:35.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:36.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:41:36.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:41:36.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:41:36.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:41:36.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:41:36.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:41:36.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:41:36.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:41:36.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:41:36.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:41:36.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:36.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:36.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:41:36.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:41:36.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:41:36.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:41:36.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:41:36.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-04-01T05:41:36.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:41:36.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:41:36.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:41:36.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:41:36.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:41:36.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:41:37.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:41:37.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-04-01T05:41:39.070 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:41:39.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:41:39.595 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:41:39.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:41:39.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:41:39.597 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.5 2024-04-01T05:41:39.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:41:39.597 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:41:39.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:41:39.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:41:39.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:41:39.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:41:39.621 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:39.621+0000 7f366fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:39.623 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:39.624+0000 7f366fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:39.625 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:39.625+0000 7f366fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:39.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:41:40.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:40.145+0000 7f366fc0d640 -1 Falling back to public interface 2024-04-01T05:41:40.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:41.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:41.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:41.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:41:41.176 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:41.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:41.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:41:41.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:42.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:42.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:42.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:41:42.474 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:42.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:42.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:41:42.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:42.532+0000 7f366fc0d640 -1 osd.5 79 log_to_monitors true 2024-04-01T05:41:42.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:41:43.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:41:43.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:41:43.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:41:43.773 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:41:43.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:41:43.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:41:44.287 INFO:tasks.workunit.client.0.smithi067.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/1499538612,v1:127.0.0.1:6843/1499538612] [v2:127.0.0.1:6844/1499538612,v1:127.0.0.1:6845/1499538612] exists,up 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:41:44.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:41:44.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:41:44.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:41:44.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:41:44.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:41:44.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:41:44.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:41:44.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:41:44.289 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:41:44.289 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:41:44.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:41:44.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:41:44.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:41:44.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:41:44.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:41:44.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:41:44.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:41:44.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:41:44.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:41:44.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:44.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:41:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-04-01T05:41:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-04-01T05:41:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-04-01T05:41:45.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:41:45.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-04-01T05:41:45.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-04-01T05:41:45.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036' 2024-04-01T05:41:45.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:41:45.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-04-01T05:41:45.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-04-01T05:41:45.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508' 2024-04-01T05:41:45.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:41:45.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645322 2024-04-01T05:41:45.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645322 2024-04-01T05:41:45.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322' 2024-04-01T05:41:45.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:41:45.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182457 2024-04-01T05:41:45.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182457 2024-04-01T05:41:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322 4-107374182457' 2024-04-01T05:41:45.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.522 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:41:45.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-04-01T05:41:45.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-04-01T05:41:45.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322 4-107374182457 5-356482285570' 2024-04-01T05:41:45.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:41:45.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855400 2024-04-01T05:41:45.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855400 2024-04-01T05:41:45.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322 4-107374182457 5-356482285570 6-150323855400' 2024-04-01T05:41:45.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.765 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:41:45.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691877 2024-04-01T05:41:45.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691877 2024-04-01T05:41:45.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322 4-107374182457 5-356482285570 6-150323855400 7-171798691877' 2024-04-01T05:41:45.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:41:45.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528349 2024-04-01T05:41:45.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528349 2024-04-01T05:41:45.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322 4-107374182457 5-356482285570 6-150323855400 7-171798691877 8-193273528349' 2024-04-01T05:41:45.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:41:45.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:41:46.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-04-01T05:41:46.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-04-01T05:41:46.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509508 3-309237645322 4-107374182457 5-356482285570 6-150323855400 7-171798691877 8-193273528349 9-330712481798' 2024-04-01T05:41:46.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:46.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-04-01T05:41:46.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:46.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:41:46.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-04-01T05:41:46.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:46.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-04-01T05:41:46.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-04-01T05:41:46.083 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836565 2024-04-01T05:41:46.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:41:46.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836565 2024-04-01T05:41:46.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:41:47.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:41:47.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:41:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836565 2024-04-01T05:41:47.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:47.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-04-01T05:41:47.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:47.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:41:47.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-04-01T05:41:47.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-04-01T05:41:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-04-01T05:41:47.694 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673036 2024-04-01T05:41:47.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:41:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673036 2024-04-01T05:41:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:47.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-04-01T05:41:47.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:47.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:41:47.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-04-01T05:41:47.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-04-01T05:41:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-04-01T05:41:47.981 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509508 2024-04-01T05:41:47.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:41:48.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509509 -lt 64424509508 2024-04-01T05:41:48.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:48.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645322 2024-04-01T05:41:48.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:48.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:41:48.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645322 2024-04-01T05:41:48.265 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:48.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645322 2024-04-01T05:41:48.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645322' 2024-04-01T05:41:48.267 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 309237645322 2024-04-01T05:41:48.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:41:48.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645323 -lt 309237645322 2024-04-01T05:41:48.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:48.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182457 2024-04-01T05:41:48.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:48.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:41:48.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182457 2024-04-01T05:41:48.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:48.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182457 2024-04-01T05:41:48.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-04-01T05:41:48.552 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182457 2024-04-01T05:41:48.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:41:48.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182457 -lt 107374182457 2024-04-01T05:41:48.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:48.835 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285570 2024-04-01T05:41:48.835 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:48.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:41:48.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285570 2024-04-01T05:41:48.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:48.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-04-01T05:41:48.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-04-01T05:41:48.838 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 356482285570 2024-04-01T05:41:48.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:41:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285570 2024-04-01T05:41:49.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:49.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855400 2024-04-01T05:41:49.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:49.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:41:49.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855400 2024-04-01T05:41:49.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855400 2024-04-01T05:41:49.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-04-01T05:41:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855400 2024-04-01T05:41:49.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:41:49.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855400 2024-04-01T05:41:49.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:49.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691877 2024-04-01T05:41:49.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:49.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:41:49.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691877 2024-04-01T05:41:49.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:49.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691877 2024-04-01T05:41:49.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-04-01T05:41:49.412 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691877 2024-04-01T05:41:49.412 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:41:49.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691877 -lt 171798691877 2024-04-01T05:41:49.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:49.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528349 2024-04-01T05:41:49.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:49.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:41:49.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528349 2024-04-01T05:41:49.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:49.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528349 2024-04-01T05:41:49.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528349' 2024-04-01T05:41:49.703 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528349 2024-04-01T05:41:49.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:41:49.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528350 -lt 193273528349 2024-04-01T05:41:49.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:41:49.986 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-04-01T05:41:49.986 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:41:49.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:41:49.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-04-01T05:41:49.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:41:49.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-04-01T05:41:49.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-04-01T05:41:49.989 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 330712481798 2024-04-01T05:41:49.989 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:41:50.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-04-01T05:41:50.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:41:50.274 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:50.274 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:50.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:41:50.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:41:50.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:41:50.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:41:50.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:41:50.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:41:50.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:41:50.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:41:50.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:41:50.908 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:41:50.908 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:41:50.908 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:41:51.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:41:51.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:51.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:51.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:51.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:41:52.130 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:52.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:52.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:52.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:52.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:52.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:52.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:20.484147+0000 '>' 2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:52.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:53.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:53.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:53.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:53.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:53.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:53.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:53.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:53.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:20.484147+0000 '>' 2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:53.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:54.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:54.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:54.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:54.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:54.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:54.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:54.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:55.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:20.484147+0000 '>' 2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:55.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:56.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:56.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:56.043 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:56.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:56.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:56.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:56.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:56.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:20.484147+0000 '>' 2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:56.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:41:57.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:41:57.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:41:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:41:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:41:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:41:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:41:57.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:41:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:52.494482+0000 '>' 2024-04-01T05:41:20.484147+0000 2024-04-01T05:41:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:41:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:41:57.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:41:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:41:57.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-04-01T05:41:59.594 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:41:59.594 INFO:tasks.workunit.client.0.smithi067.stdout:hinfo_key 2024-04-01T05:41:59.594 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:41:59.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:41:59.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:41:59.871 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.9 2024-04-01T05:41:59.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:41:59.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:41:59.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:41:59.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:41:59.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:41:59.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:41:59.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:59.900+0000 7f60fcc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:59.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:59.902+0000 7f60fcc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:41:59.903 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:41:59.903+0000 7f60fcc0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:00.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:42:00.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:00.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:42:00.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:00.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:00.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:00.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:00.166 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:42:00.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:00.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:00.428 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:42:00.428+0000 7f60fcc0c640 -1 Falling back to public interface 2024-04-01T05:42:00.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:01.448 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:42:01.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:01.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:01.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:02.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:42:02.631+0000 7f60fcc0c640 -1 osd.9 84 log_to_monitors true 2024-04-01T05:42:02.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:02.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:02.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:02.758 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:42:02.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:02.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:03.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:04.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:04.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:04.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:04.057 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:42:04.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:04.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:04.538 INFO:tasks.workunit.client.0.smithi067.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/3824042134,v1:127.0.0.1:6875/3824042134] [v2:127.0.0.1:6876/3824042134,v1:127.0.0.1:6877/3824042134] exists,up 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:42:04.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:04.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:04.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:04.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:04.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:04.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:04.623 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:05.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:05.009 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:4 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:5 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:6 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:7 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:8 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:9' 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-04-01T05:42:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-04-01T05:42:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-04-01T05:42:05.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:05.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-04-01T05:42:05.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-04-01T05:42:05.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041' 2024-04-01T05:42:05.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.247 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:05.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509513 2024-04-01T05:42:05.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509513 2024-04-01T05:42:05.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513' 2024-04-01T05:42:05.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645327 2024-04-01T05:42:05.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645327 2024-04-01T05:42:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327' 2024-04-01T05:42:05.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-04-01T05:42:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-04-01T05:42:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327 4-107374182462' 2024-04-01T05:42:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:05.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285575 2024-04-01T05:42:05.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285575 2024-04-01T05:42:05.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327 4-107374182462 5-356482285575' 2024-04-01T05:42:05.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:05.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855405 2024-04-01T05:42:05.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855405 2024-04-01T05:42:05.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327 4-107374182462 5-356482285575 6-150323855405' 2024-04-01T05:42:05.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:05.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-04-01T05:42:05.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-04-01T05:42:05.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327 4-107374182462 5-356482285575 6-150323855405 7-171798691882' 2024-04-01T05:42:05.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:05.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-04-01T05:42:05.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-04-01T05:42:05.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327 4-107374182462 5-356482285575 6-150323855405 7-171798691882 8-193273528354' 2024-04-01T05:42:05.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:05.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:42:06.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-04-01T05:42:06.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-04-01T05:42:06.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673041 2-64424509513 3-309237645327 4-107374182462 5-356482285575 6-150323855405 7-171798691882 8-193273528354 9-377957122050' 2024-04-01T05:42:06.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:06.089 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-04-01T05:42:06.089 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:06.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:42:06.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-04-01T05:42:06.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:06.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-04-01T05:42:06.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-04-01T05:42:06.092 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836570 2024-04-01T05:42:06.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836570 2024-04-01T05:42:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:42:07.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:42:07.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:07.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836570 2024-04-01T05:42:07.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:07.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-04-01T05:42:07.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:07.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:42:07.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-04-01T05:42:07.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:07.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-04-01T05:42:07.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-04-01T05:42:07.689 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949673041 2024-04-01T05:42:07.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:42:07.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-04-01T05:42:07.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:07.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509513 2024-04-01T05:42:07.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:07.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:42:07.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509513 2024-04-01T05:42:07.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:07.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509513 2024-04-01T05:42:07.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-04-01T05:42:07.977 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509513 2024-04-01T05:42:07.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:42:08.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509514 -lt 64424509513 2024-04-01T05:42:08.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:08.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645327 2024-04-01T05:42:08.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:08.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:42:08.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645327 2024-04-01T05:42:08.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:08.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645327 2024-04-01T05:42:08.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645327' 2024-04-01T05:42:08.269 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.3 seq 309237645327 2024-04-01T05:42:08.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:42:08.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645328 -lt 309237645327 2024-04-01T05:42:08.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:08.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-04-01T05:42:08.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:08.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:42:08.552 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-04-01T05:42:08.553 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:08.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-04-01T05:42:08.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-04-01T05:42:08.554 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.4 seq 107374182462 2024-04-01T05:42:08.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:42:08.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182462 -lt 107374182462 2024-04-01T05:42:08.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285575 2024-04-01T05:42:08.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:08.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:42:08.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285575 2024-04-01T05:42:08.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:08.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285575 2024-04-01T05:42:08.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285575' 2024-04-01T05:42:08.841 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.5 seq 356482285575 2024-04-01T05:42:08.842 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:42:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285576 -lt 356482285575 2024-04-01T05:42:09.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:09.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855405 2024-04-01T05:42:09.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:09.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:42:09.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855405 2024-04-01T05:42:09.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:09.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855405 2024-04-01T05:42:09.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-04-01T05:42:09.137 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.6 seq 150323855405 2024-04-01T05:42:09.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:42:09.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855405 2024-04-01T05:42:09.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:09.454 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-04-01T05:42:09.454 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:09.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:42:09.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-04-01T05:42:09.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:09.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-04-01T05:42:09.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-04-01T05:42:09.457 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.7 seq 171798691882 2024-04-01T05:42:09.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:42:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691882 -lt 171798691882 2024-04-01T05:42:09.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:09.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-04-01T05:42:09.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:09.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:42:09.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-04-01T05:42:09.742 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:09.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-04-01T05:42:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-04-01T05:42:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.8 seq 193273528354 2024-04-01T05:42:09.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:42:10.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528355 -lt 193273528354 2024-04-01T05:42:10.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:10.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-04-01T05:42:10.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:10.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:42:10.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-04-01T05:42:10.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:10.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-04-01T05:42:10.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-04-01T05:42:10.028 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.9 seq 377957122050 2024-04-01T05:42:10.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:42:10.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-04-01T05:42:10.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:42:10.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:10.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:10.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:42:10.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:42:10.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:42:10.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:42:10.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:10.946 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:42:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:42:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:42:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:42:11.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:42:11.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-04-01T05:42:11.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:42:11.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:42:11.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-04-01T05:42:11.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269075"' 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269075' 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269076"' 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269076' 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:42:11.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 269075 269076' 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269075 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269078: /' 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:42:11.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269080: /' 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:42:14.355 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:42:14.356 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:42:14.357 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:42:14.358 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:14.359 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: start osd.5 2024-04-01T05:42:14.360 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:42:14.447 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-04-01T05:42:14.448 INFO:tasks.workunit.client.0.smithi067.stderr:269080: remove 2#2:eb822e21:::SOMETHING:head# 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:42:14.449 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:42:14.450 INFO:tasks.workunit.client.0.smithi067.stderr:269080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:42:14.451 INFO:tasks.workunit.client.0.smithi067.stderr:269080: start osd.9 2024-04-01T05:42:14.452 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2024-04-01T05:42:14.477+0000 7efce0215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2024-04-01T05:42:14.479+0000 7efce0215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2024-04-01T05:42:14.480+0000 7efce0215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:19.175 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 0 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2024-04-01T05:42:14.997+0000 7efce0215640 -1 Falling back to public interface 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:19.176 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 1 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2024-04-01T05:42:17.851+0000 7efce0215640 -1 osd.9 89 log_to_monitors true 2024-04-01T05:42:19.177 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 3 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269080: osd.9 up in weight 1 up_from 94 up_thru 0 down_at 90 last_c269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2024-04-01T05:42:14.387+0000 7ffbda618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2024-04-01T05:42:14.389+0000 7ffbda618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2024-04-01T05:42:14.390+0000 7ffbda618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:19.178 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 0 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2024-04-01T05:42:14.910+0000 7ffbda618640 -1 Falling back to public interface 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.179 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 1 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2024-04-01T05:42:17.715+0000 7ffbda618640 -1 osd.5 89 log_to_monitors true 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:19.180 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:19.181 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:19.181 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 3 2024-04-01T05:42:19.181 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:19.181 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269078: osd.5 up in weight 1 up_from 94 up_thru 83 down_at 90 last_lean_interval [88,89) [v2:127.0.0.1:6874/1917675800,v1:127.0.0.1:6875/1917675800] [v2:127.0.0.1:6876/1917675800,v1:127.0.0.1:6877/1917675800] exists,up 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:19.896 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 1 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 2 2024-04-01T05:42:19.897 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 3 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 4 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 5 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 6 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 7 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 8 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: 9' 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836574 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836574 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574' 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:19.898 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045' 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qaclean_interval [83,89) [v2:127.0.0.1:6842/3000535156,v1:127.0.0.1:6843/3000535156] [v2:127.0.0.1:6844/3000535156,v1:127.0.0.1:6845/3000535156] exists,up 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 1 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 2 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 3 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 4 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 5 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 6 2024-04-01T05:42:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 7 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 8 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: 9' 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046' 2024-04-01T05:42:19.901 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/q/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517' 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645332 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645332 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332' 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:21.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332 4-107374182467' 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332 4-107374182467 5-403726925827' 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855411 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855411 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332 4-107374182467 5-403726925827 6-150323855411' 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.157 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332 4-107374182467 5-403726925827 6-150323855411 7-171798691887' 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clonea/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509518 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509518 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518' 2024-04-01T05:42:21.158 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645331 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645331 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331' 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182466 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182466 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331 4-107374182466' 2024-04-01T05:42:21.159 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331 4-107374182466 5-403726925826' 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331 4-107374182466 5-403726925826 6-150323855410' 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691886 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331 4-107374182466 5-403726925826 6-150323855410 7-171798691886' 2024-04-01T05:42:21.160 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.161 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:21.877 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-04-01T05:42:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-04-01T05:42:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331 4-107374182466 5-403726925826 6-150323855410 7-171798691886 8-193273528359' 2024-04-01T05:42:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:42:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-04-01T05:42:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509518 3-309237645331 4-107374182466 5-403726925826 6-150323855410 7-171798691886 8-193273528359 9-403726925826' 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-04-01T05:42:21.879 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.0 seq 21474836575 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836575 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.1 seq 42949673046 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509518 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:21.881 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone..client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332 4-107374182467 5-403726925827 6-150323855411 7-171798691887 8-193273528358' 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509517 3-309237645332 4-107374182467 5-403726925827 6-150323855411 7-171798691887 8-193273528358 9-403726925827' 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836574 2024-04-01T05:42:21.882 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836574 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836574 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.0 seq 21474836574 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836574 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-04-01T05:42:21.883 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.1 seq 42949673045 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673045 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:21.884 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:42:22.753 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.cclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509518 2024-04-01T05:42:22.753 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.753 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509518 2024-04-01T05:42:22.753 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.2 seq 64424509518 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509518 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645331 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645331 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645331 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645331' 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.3 seq 309237645331 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:42:22.754 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645331 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182466 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182466 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182466 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182466' 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.4 seq 107374182466 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182467 -lt 107374182466 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-403726925826 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:22.755 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:42:22.756 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-403726925826 2024-04-01T05:42:22.756 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.756 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.2 seq 64424509517 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:42:22.821 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509517 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645332 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645332 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645332 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645332' 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.3 seq 309237645332 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645332 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:22.822 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.4 seq 107374182467 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182467 -lt 107374182467 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-403726925827 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:42:22.823 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-403726925827 2024-04-01T05:42:22.824 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:22.824 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-04-01T05:42:24.011 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitiing osd.5 seq 403726925826' 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.5 seq 403726925826 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:24.012 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-04-01T05:42:24.013 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-04-01T05:42:24.013 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.6 seq 150323855410 2024-04-01T05:42:24.013 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:42:24.013 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855411 -lt 150323855410 2024-04-01T05:42:24.013 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:24.013 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.7 seq 171798691886 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691886 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-04-01T05:42:24.014 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.8 seq 193273528359 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:42:24.015 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528359 2024-04-01T05:42:24.153 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_png osd.5 seq 403726925827' 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.5 seq 403726925827 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855411 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855411 2024-04-01T05:42:24.154 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855411 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.6 seq 150323855411 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855411 -lt 150323855411 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.7 seq 171798691887 2024-04-01T05:42:24.155 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691887 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.8 seq 193273528358 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:42:24.156 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528359 -lt 193273528358 2024-04-01T05:42:25.321 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgg_stats: for s in $seqs 2024-04-01T05:42:25.322 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-403726925826 2024-04-01T05:42:25.322 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:25.322 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:42:25.322 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-403726925826 2024-04-01T05:42:25.322 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 403726925826' 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: waiting osd.9 seq 403726925826 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:42:25.323 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:42:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:42:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:42:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:42:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:269078: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:25.326 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:42:25.326 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:42:25.326 INFO:tasks.workunit.client.0.smithi067.stderr:269078: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:42:25.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:42:25.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:42:25.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269076 2024-04-01T05:42:25.537 INFO:tasks.workunit.client.0.smithi067.stderr:_stats: for s in $seqs 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-403726925827 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-403726925827 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 403726925827' 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: waiting osd.9 seq 403726925827 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:25.538 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:269080: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:42:25.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:42:25.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:42:25.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:25.837 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:26.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:26.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:42:26.390 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:42:26.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:26.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:26.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:26.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:52.494482+0000 '>' 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:26.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:42:27.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:42:27.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:42:27.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:42:27.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:27.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:27.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:27.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:27.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:52.494482+0000 '>' 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:27.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:42:28.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:42:28.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:42:28.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:42:28.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:28.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:28.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:28.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:29.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:52.494482+0000 '>' 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:29.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:42:30.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:42:30.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:42:30.281 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:42:30.281 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:30.281 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:30.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:30.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:30.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:52.494482+0000 '>' 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:30.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:42:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:42:31.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:42:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:42:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:31.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:31.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:41:52.494482+0000 '>' 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:31.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:42:32.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:42:32.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:42:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:42:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:42:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:42:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:42:32.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:42:33.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:42:27.373653+0000 '>' 2024-04-01T05:41:52.494482+0000 2024-04-01T05:42:33.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:42:33.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:42:33.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:42:33.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:42:33.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:42:33.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272506"' 2024-04-01T05:42:33.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272506' 2024-04-01T05:42:33.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272508"' 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272508' 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:42:33.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 272506 272508' 2024-04-01T05:42:33.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:42:33.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:42:33.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272506 2024-04-01T05:42:33.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272509: /' 2024-04-01T05:42:33.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:42:33.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272511: /' 2024-04-01T05:42:36.012 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.012 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:42:36.012 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:42:36.012 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:42:36.012 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:42:36.012 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.013 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: _ 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: hinfo_key 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: snapset 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:42:36.014 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:42:36.015 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:42:36.015 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:42:36.015 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:42:36.015 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:42:36.015 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:42:36.016 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:42:36.017 INFO:tasks.workunit.client.0.smithi067.stderr:272509: start osd.5 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:42:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: _ 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: hinfo_key 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: snapset 2024-04-01T05:42:36.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:42:36.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:42:36.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:42:36.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:42:36.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-04-01T05:42:36.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-04-01T05:42:36.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:42:36.547 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-04-01T05:42:36.548 INFO:tasks.workunit.client.0.smithi067.stderr:272511: start osd.9 2024-04-01T05:42:40.491 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2024-04-01T05:42:36.044+0000 7f7fecc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2024-04-01T05:42:36.046+0000 7f7fecc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2024-04-01T05:42:36.047+0000 7f7fecc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:40.492 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 0 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2024-04-01T05:42:36.576+0000 7f7fecc0d640 -1 Falling back to public interface 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 1 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:40.493 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2024-04-01T05:42:39.124+0000 7f7fecc0d640 -1 osd.5 96 log_to_monitors true 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 3 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: osd.5 up in weight 1 up_from 100 up_thru 94 down_overy_ops 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2024-04-01T05:42:36.570+0000 7f49688b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2024-04-01T05:42:36.572+0000 7f49688b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2024-04-01T05:42:36.573+0000 7f49688b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:41.312 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 0 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2024-04-01T05:42:37.093+0000 7f49688b1640 -1 Falling back to public interface 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:41.313 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 1 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2024-04-01T05:42:39.939+0000 7f49688b1640 -1 osd.9 96 log_to_monitors true 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:41.314 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:41.315 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 3 2024-04-01T05:42:41.315 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:41.315 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-04-01T05:42:41.585 INFO:tasks.workunit.client.0.smithi067.stderr:272511: osd.9 up in weight 1 up_from 102 up_thru 0 down_aat 97 last_clean_interval [94,96) [v2:127.0.0.1:6842/607558725,v1:127.0.0.1:6843/607558725] [v2:127.0.0.1:6844/607558725,v1:127.0.0.1:6845/607558725] exists,up 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:42:41.585 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:41.585 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:41.585 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:41.585 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:41.585 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:41.586 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 1 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 2 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 3 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 4 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 5 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 6 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 7 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 8 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: 9' 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:41.587 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051' 2024-04-01T05:42:41.588 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:42.295 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.clit 97 last_clean_interval [94,96) [v2:127.0.0.1:6874/3025900395,v1:127.0.0.1:6875/3025900395] [v2:127.0.0.1:6876/3025900395,v1:127.0.0.1:6877/3025900395] exists,up 0726d7fc-af3e-41b8-b7d7-d599ec6371e9 2024-04-01T05:42:42.295 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:42.295 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:42.295 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:42.295 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:42.295 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:42.296 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 1 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 2 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 3 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 4 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 5 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 6 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 7 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 8 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: 9' 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:42.297 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052' 2024-04-01T05:42:42.298 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.ent.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524' 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645338 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645338 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338' 2024-04-01T05:42:43.253 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338 4-107374182472' 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338 4-107374182472 5-429496729602' 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855416 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855416 2024-04-01T05:42:43.254 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338 4-107374182472 5-429496729602 6-150323855416' 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338 4-107374182472 5-429496729602 6-150323855416 7-171798691892' 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.255 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephteclient.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525' 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645339 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645339 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339' 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-04-01T05:42:43.781 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339 4-107374182473' 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729603 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729603 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339 4-107374182473 5-429496729603' 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339 4-107374182473 5-429496729603 6-150323855417' 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:43.782 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691893 2024-04-01T05:42:43.783 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-04-01T05:42:43.783 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339 4-107374182473 5-429496729603 6-150323855417 7-171798691893' 2024-04-01T05:42:43.783 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:43.783 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cepst/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338 4-107374182472 5-429496729602 6-150323855416 7-171798691892 8-193273528364' 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664194 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664194 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673051 2-64424509524 3-309237645338 4-107374182472 5-429496729602 6-150323855416 7-171798691892 8-193273528364 9-438086664194' 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-04-01T05:42:44.063 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.0 seq 21474836581 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836581 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:44.064 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.1 seq 42949673051 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673051 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:44.065 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:42:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtesthtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-04-01T05:42:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-04-01T05:42:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339 4-107374182473 5-429496729603 6-150323855417 7-171798691893 8-193273528366' 2024-04-01T05:42:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:42:44.544 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664195 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664195 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673052 2-64424509525 3-309237645339 4-107374182473 5-429496729603 6-150323855417 7-171798691893 8-193273528366 9-438086664195' 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.0 seq 21474836582 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:44.545 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.1 seq 42949673052 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:44.546 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cepht/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.2 seq 64424509524 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509524 2024-04-01T05:42:45.049 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645338 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645338 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645338 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645338' 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.3 seq 309237645338 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645338 -lt 309237645338 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-04-01T05:42:45.050 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.4 seq 107374182472 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182472 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-429496729602 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-429496729602 2024-04-01T05:42:45.051 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.052 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-04-01T05:42:45.496 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.2 seq 64424509525 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645339 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645339 2024-04-01T05:42:45.497 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645339 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645339' 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.3 seq 309237645339 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645339 -lt 309237645339 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-04-01T05:42:45.498 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.4 seq 107374182473 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182473 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-429496729603 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-429496729603 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:45.499 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729603 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: o 'waiting osd.5 seq 429496729602' 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.5 seq 429496729602 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729602 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855416 2024-04-01T05:42:46.310 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855416 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855416 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.6 seq 150323855416 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855416 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-04-01T05:42:46.311 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.7 seq 171798691892 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691892 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.8 seq 193273528364 2024-04-01T05:42:46.312 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:42:46.313 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528364 2024-04-01T05:42:46.771 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: echo 'waiting osd.5 seq 429496729603' 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.5 seq 429496729603 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729603 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.6 seq 150323855417 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855417 2024-04-01T05:42:46.772 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.7 seq 171798691893 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691893 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-04-01T05:42:46.773 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:46.774 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-04-01T05:42:46.774 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-04-01T05:42:46.774 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.8 seq 193273528366 2024-04-01T05:42:46.774 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:42:46.774 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528366 -lt 193273528366 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221flush_pg_stats: for s in $seqs 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664194 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664194 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664194 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664194' 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: waiting osd.9 seq 438086664194 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664195 -lt 438086664194 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:42:47.624 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:42:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:42:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:272509: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:42:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:42:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:42:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272508 2024-04-01T05:42:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:0: flush_pg_stats: for s in $seqs 2024-04-01T05:42:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664195 2024-04-01T05:42:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:42:48.065 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664195 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664195 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664195' 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: waiting osd.9 seq 438086664195 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664196 -lt 438086664195 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:42:48.066 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:272511: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:42:48.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:42:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:42:48.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:42:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:42:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:42:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:42:48.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:42:48.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:42:48.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-04-01T05:42:48.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 275739"' 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 275739' 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 275741"' 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 275741' 2024-04-01T05:42:48.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 275739 275741' 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/275742: /' 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 275739 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:48.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/275744: /' 2024-04-01T05:42:51.171 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.171 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:42:51.172 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: remove 1#2:eb822e21:::SOMETHING:head# 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:42:51.173 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:42:51.174 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:42:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:275744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: start osd.5 2024-04-01T05:42:51.176 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:42:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:42:51.285 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:42:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: remove 0#2:eb822e21:::SOMETHING:head# 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:42:51.287 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:42:51.288 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:42:51.289 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: start osd.3 2024-04-01T05:42:51.290 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:42:55.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:42:55.834 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:42:55.834 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:42:55.834 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2024-04-01T05:42:51.204+0000 7fdd82fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2024-04-01T05:42:51.207+0000 7fdd82fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2024-04-01T05:42:51.208+0000 7fdd82fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:55.835 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 0 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2024-04-01T05:42:51.732+0000 7fdd82fff640 -1 Falling back to public interface 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 1 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:55.836 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2024-04-01T05:42:54.524+0000 7fdd82fff640 -1 osd.5 104 log_to_monitors true 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 3 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:42:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:275744: osd.5 up in weight 1 up_from 108 up_thru 108 down_at 105 l275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:42:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:42:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:42:56.022 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2024-04-01T05:42:51.314+0000 7f0018f9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2024-04-01T05:42:51.317+0000 7f0018f9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2024-04-01T05:42:51.318+0000 7f0018f9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 0 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:56.023 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2024-04-01T05:42:51.834+0000 7f0018f9a640 -1 Falling back to public interface 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 1 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2 2024-04-01T05:42:56.024 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2024-04-01T05:42:54.655+0000 7f0018f9a640 -1 osd.3 104 log_to_monitors true 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 3 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:42:56.025 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:42:56.890 INFO:tasks.workunit.client.0.smithi067.stderr:275742: osd.3 up in weight 1 up_from 109 up_thru 102 down_at 105 last_clean_interval [100,104) [v2:127.0.0.1:6826/2211910127,v1:127.0.0.1:6827/2211910127] [v2:127.0.0.1:6828/2211910127,v1:127.0.0.1:6829/2211910127] exists,up 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:56.891 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 1 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 2 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 3 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 4 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 5 2024-04-01T05:42:56.892 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 6 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 7 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 8 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: 9' 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-04-01T05:42:56.893 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056' 2024-04-01T05:42:56.894 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.clieast_clean_interval [72,104) [v2:127.0.0.1:6842/1944445940,v1:127.0.0.1:6843/1944445940] [v2:127.0.0.1:6844/1944445940,v1:127.0.0.1:6845/1944445940] exists,up f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:42:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:42:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:42:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:42:57.146 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:42:57.147 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 1 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 2 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 3 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 4 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 5 2024-04-01T05:42:57.148 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 6 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 7 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 8 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: 9' 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.149 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:42:57.150 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-04-01T05:42:57.150 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-04-01T05:42:57.150 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057' 2024-04-01T05:42:57.150 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.952 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.cliennt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529' 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=468151435266 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 468151435266 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266' 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-04-01T05:42:57.953 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266 4-107374182477' 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266 4-107374182477 5-463856467970' 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266 4-107374182477 5-463856467970 6-150323855421' 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-04-01T05:42:57.954 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-04-01T05:42:57.955 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266 4-107374182477 5-463856467970 6-150323855421 7-171798691897' 2024-04-01T05:42:57.955 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:57.955 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:58.063 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:42:58.063 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530' 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=468151435267 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 468151435267 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267' 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267 4-107374182478' 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:58.064 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467971 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467971 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267 4-107374182478 5-463856467971' 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855422 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855422 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267 4-107374182478 5-463856467971 6-150323855422' 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267 4-107374182478 5-463856467971 6-150323855422 7-171798691898' 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:58.065 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:42:59.966 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtestt/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-04-01T05:42:59.966 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-04-01T05:42:59.966 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266 4-107374182477 5-463856467970 6-150323855421 7-171798691897 8-193273528369' 2024-04-01T05:42:59.966 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:42:59.966 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:42:59.966 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664199 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664199 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673056 2-64424509529 3-468151435266 4-107374182477 5-463856467970 6-150323855421 7-171798691897 8-193273528369 9-438086664199' 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.0 seq 21474836586 2024-04-01T05:42:59.967 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836586 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836586 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:42:59.968 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-04-01T05:42:59.969 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-04-01T05:42:59.969 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.1 seq 42949673056 2024-04-01T05:42:59.969 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:42:59.969 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673056 2024-04-01T05:43:00.158 INFO:tasks.workunit.client.0.smithi067.stderr:27/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528370 2024-04-01T05:43:00.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528370 2024-04-01T05:43:00.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267 4-107374182478 5-463856467971 6-150323855422 7-171798691898 8-193273528370' 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664200 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664200 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509530 3-468151435267 4-107374182478 5-463856467971 6-150323855422 7-171798691898 8-193273528370 9-438086664200' 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-04-01T05:43:00.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.0 seq 21474836587 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836587 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836587 -lt 21474836587 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-04-01T05:43:00.160 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.1 seq 42949673057 2024-04-01T05:43:00.161 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:43:00.161 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673057 2024-04-01T05:43:00.912 INFO:tasks.workunit.client.0.smithi067.stderr:2755744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.2 seq 64424509529 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509529 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-468151435266 2024-04-01T05:43:00.913 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-468151435266 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=468151435266 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 468151435266' 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.3 seq 468151435266 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 468151435267 -lt 468151435266 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-04-01T05:43:00.914 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.4 seq 107374182477 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182477 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-463856467970 2024-04-01T05:43:00.915 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.156 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.2 seq 64424509530 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-468151435267 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.157 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-468151435267 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=468151435267 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 468151435267' 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.3 seq 468151435267 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 468151435267 -lt 468151435267 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.158 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.4 seq 107374182478 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182478 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-463856467971 2024-04-01T05:43:01.159 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22211: flush_pg_stats: osd=5 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-463856467970 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 463856467970' 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.5 seq 463856467970 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467972 -lt 463856467970 2024-04-01T05:43:01.815 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.6 seq 150323855421 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855421 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-04-01T05:43:01.816 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.7 seq 171798691897 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691897 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:01.817 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.c11: flush_pg_stats: osd=5 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-463856467971 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467971 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 463856467971' 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.5 seq 463856467971 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467972 -lt 463856467971 2024-04-01T05:43:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855422 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855422 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855422 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.6 seq 150323855422 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855422 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.7 seq 171798691898 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691898 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528370 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528370 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528370 2024-04-01T05:43:03.533 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.cllient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.8 seq 193273528369 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528369 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664199 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664199 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664199 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664199' 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: waiting osd.9 seq 438086664199 2024-04-01T05:43:03.534 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664200 -lt 438086664199 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:43:03.535 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:275744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:43:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:43:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:275744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:43:03.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:ient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.8 seq 193273528370 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528371 -lt 193273528370 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664200 2024-04-01T05:43:03.688 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664200 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664200 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664200' 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: waiting osd.9 seq 438086664200 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664200 -lt 438086664200 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:43:03.689 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:275742: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 275741 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:43:03.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-04-01T05:43:03.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:43:04.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:43:04.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:43:04.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:42:27.373653+0000 2024-04-01T05:43:04.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T05:43:04.565 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-04-01T05:43:04.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T05:42:27.373653+0000 2024-04-01T05:43:04.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T05:43:04.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:42:27.373653+0000 2024-04-01T05:43:04.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:43:04.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:43:04.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:42:27.373653+0000 '>' 2024-04-01T05:42:27.373653+0000 2024-04-01T05:43:04.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:43:05.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:43:05.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:43:05.886 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:43:05.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:43:05.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:43:05.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:43:05.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:43:06.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:42:27.373653+0000 '>' 2024-04-01T05:42:27.373653+0000 2024-04-01T05:43:06.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:43:07.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:43:07.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:43:07.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T05:43:07.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T05:43:07.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:43:07.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:43:07.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T05:43:07.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:43:04.910938+0000 '>' 2024-04-01T05:42:27.373653+0000 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 279141"' 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 279141' 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 279143"' 2024-04-01T05:43:07.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 279143' 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 279141 279143' 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/279144: /' 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 279141 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:43:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/279146: /' 2024-04-01T05:43:10.338 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.338 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:43:10.338 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:43:10.339 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: _ 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: hinfo_key 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: snapset 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-04-01T05:43:10.340 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:43:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-01T05:43:10.343 INFO:tasks.workunit.client.0.smithi067.stderr:279144: start osd.3 2024-04-01T05:43:10.752 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-04-01T05:43:10.753 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:43:10.754 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: _ 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: hinfo_key 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: snapset 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:43:10.755 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:43:10.756 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:43:10.757 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:43:10.758 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-04-01T05:43:10.758 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-04-01T05:43:10.758 INFO:tasks.workunit.client.0.smithi067.stderr:279146: start osd.5 2024-04-01T05:43:14.500 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:43:14.500 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2024-04-01T05:43:10.371+0000 7ffb96864640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2024-04-01T05:43:10.373+0000 7ffb96864640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2024-04-01T05:43:10.375+0000 7ffb96864640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:43:14.501 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 0 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2024-04-01T05:43:10.889+0000 7ffb96864640 -1 Falling back to public interface 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 1 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:14.502 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2024-04-01T05:43:13.650+0000 7ffb96864640 -1 osd.3 110 log_to_monitors true 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 3 2024-04-01T05:43:14.503 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-04-01T05:43:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-04-01T05:43:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-04-01T05:43:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2024-04-01T05:43:10.782+0000 7f087c2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2024-04-01T05:43:10.784+0000 7f087c2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2024-04-01T05:43:10.785+0000 7f087c2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:43:15.411 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 0 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2024-04-01T05:43:11.300+0000 7f087c2d9640 -1 Falling back to public interface 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 1 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:43:15.412 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2024-04-01T05:43:13.945+0000 7f087c2d9640 -1 osd.5 110 log_to_monitors true 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 3 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:15.413 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279146: osd.5 up in weight 1 up_from 115 up_thru 108 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279144: osd.3 up in weight 1 up_from 114 up_thru 114 down_at 111 last_clean_interval [109,110) [v2:127.0.0.1:6826/186967967,v1:127.0.0.1:6827/186967967] [v2:127.0.0.1:6828/186967967,v1:127.0.0.1:6829/186967967] exists,up f106e71b-2872-44b6-86db-49f229aa28e9 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:43:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:43:15.775 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 1 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 2 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 3 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 4 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 5 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 6 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 7 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 8 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: 9' 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836592 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836592 2024-04-01T05:43:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592' 2024-04-01T05:43:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:43:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-04-01T05:43:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-04-01T05:43:15.777 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062' 2024-04-01T05:43:16.486 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/n_at 111 last_clean_interval [108,110) [v2:127.0.0.1:6842/149997727,v1:127.0.0.1:6843/149997727] [v2:127.0.0.1:6844/149997727,v1:127.0.0.1:6845/149997727] exists,up 5306857f-96f9-4e56-bb5e-552f11be76f2 2024-04-01T05:43:16.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:43:16.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:43:16.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:43:16.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:43:16.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:43:16.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 1 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 2 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 3 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 4 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 5 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 6 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 7 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 8 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: 9' 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836593 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836593 2024-04-01T05:43:16.488 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593' 2024-04-01T05:43:16.489 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:16.489 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:43:16.489 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-04-01T05:43:16.489 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-04-01T05:43:16.489 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063' 2024-04-01T05:43:16.489 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clonstandalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534' 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.018 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746' 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182483 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182483 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746 4-107374182483' 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746 4-107374182483 5-493921239042' 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:43:17.019 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746 4-107374182483 5-493921239042 6-150323855427' 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691903 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746 4-107374182483 5-493921239042 6-150323855427 7-171798691903' 2024-04-01T05:43:17.020 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.485 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/ste.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:43:17.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-04-01T05:43:17.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-04-01T05:43:17.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536' 2024-04-01T05:43:17.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271747 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271747 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747' 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182484 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182484 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747 4-107374182484' 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239043 2024-04-01T05:43:17.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239043 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747 4-107374182484 5-493921239043' 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855428 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855428 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747 4-107374182484 5-493921239043 6-150323855428' 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747 4-107374182484 5-493921239043 6-150323855428 7-171798691904' 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.487 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:43:17.874 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/candalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746 4-107374182483 5-493921239042 6-150323855427 7-171798691903 8-193273528375' 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664205 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664205 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509534 3-489626271746 4-107374182483 5-493921239042 6-150323855427 7-171798691903 8-193273528375 9-438086664205' 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836592 2024-04-01T05:43:17.875 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836592 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836592 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.0 seq 21474836592 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836592 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-04-01T05:43:17.876 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.1 seq 42949673062 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673062 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-04-01T05:43:17.877 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtesephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747 4-107374182484 5-493921239043 6-150323855428 7-171798691904 8-193273528376' 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664206 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664206 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836593 1-42949673063 2-64424509536 3-489626271747 4-107374182484 5-493921239043 6-150323855428 7-171798691904 8-193273528376 9-438086664206' 2024-04-01T05:43:18.262 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836593 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836593 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836593 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.0 seq 21474836593 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836593 -lt 21474836593 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.263 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.1 seq 42949673063 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673063 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:18.264 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-04-01T05:43:18.265 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.265 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cept/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.2 seq 64424509534 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509534 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:18.966 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271746 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271746 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.3 seq 489626271746 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271746 -lt 489626271746 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182483 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182483 2024-04-01T05:43:18.967 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182483 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.4 seq 107374182483 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182483 -lt 107374182483 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-493921239042 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-493921239042 2024-04-01T05:43:18.968 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.156 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-04-01T05:43:20.156 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 493921239042' 2024-04-01T05:43:20.156 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.5 seq 493921239042 2024-04-01T05:43:20.156 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239042 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.6 seq 150323855427 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855427 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:20.157 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.7 seq 171798691903 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691903 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-04-01T05:43:20.158 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.8 seq 193273528375 2024-04-01T05:43:20.159 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 19327htest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.2 seq 64424509536 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509536 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271747 2024-04-01T05:43:20.606 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271747 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271747 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.3 seq 489626271747 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271746 -lt 489626271747 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271747 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182484 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:20.607 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182484 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182484 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.4 seq 107374182484 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182484 -lt 107374182484 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-493921239043 2024-04-01T05:43:20.608 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:21.483 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-04-01T05:43:21.483 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-493921239043 2024-04-01T05:43:21.483 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239043 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 493921239043' 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.5 seq 493921239043 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239043 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855428 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855428 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855428 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-04-01T05:43:21.484 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.6 seq 150323855428 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855428 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.7 seq 171798691904 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691904 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:21.485 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-04-01T05:43:21.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:21.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-04-01T05:43:21.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-04-01T05:43:21.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:21.486 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-04-01T05:43:21.627 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest3528377 -lt 193273528375 2024-04-01T05:43:21.627 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:21.627 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664205 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664205 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664205 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664205' 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: waiting osd.9 seq 438086664205 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664206 -lt 438086664205 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:21.628 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:21.629 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:43:21.630 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:43:21.630 INFO:tasks.workunit.client.0.smithi067.stderr:279144: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:43:21.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:43:21.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-04-01T05:43:21.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 279143 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.8 seq 193273528376 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528377 -lt 193273528376 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-438086664206 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:43:23.112 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-438086664206 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664206 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 438086664206' 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: waiting osd.9 seq 438086664206 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664206 -lt 438086664206 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:43:23.113 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:279146: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-04-01T05:43:23.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-04-01T05:43:23.115 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-04-01T05:43:23.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-04-01T05:43:23.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-04-01T05:43:23.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-04-01T05:43:23.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:43:23.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:43:23.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:43:23.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:43:23.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:43:23.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:43:23.171 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:43:23.171 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:43:23.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:43:23.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:43:23.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:43:23.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:43:23.279 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:43:23.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:43:23.280 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:43:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:43:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:43:23.281 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:43:23.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:43:23.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:43:23.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:43:23.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:43:23.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:43:23.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:43:23.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:43:23.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:43:23.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:23.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:23.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:43:23.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:43:23.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:43:23.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:43:23.349 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:43:23.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:43:23.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:43:23.351 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:43:23.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:43:23.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:43:23.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:43:23.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:43:23.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:43:23.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:43:23.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:43:23.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:43:23.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:23.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:23.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:43:23.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:43:23.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:43:23.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:43:23.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:43:23.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:23.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:23.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:43:23.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:43:23.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:43:23.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-04-01T05:43:23.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:23.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-04-01T05:43:23.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:43:23.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:43:23.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:43:23.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:43:23.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:43:23.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:43:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:43:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:43:23.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:43:24.005 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:43:24.006 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:43:24.006 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:24.006 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:24.006 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:43:24.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:43:24.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:43:24.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-04-01T05:43:24.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:43:24.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:43:24.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:43:24.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:43:24.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:43:24.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:43:24.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:43:24.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:43:24.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:43:24.904 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:43:24.905 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:24.905 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:24.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:24.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:43:24.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:43:24.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-04-01T05:43:24.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:43:24.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:43:24.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:43:24.938 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:43:24.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ca72484-20ba-49bb-bc41-e7fcc59d5854 2024-04-01T05:43:24.939 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 3ca72484-20ba-49bb-bc41-e7fcc59d5854 2024-04-01T05:43:24.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3ca72484-20ba-49bb-bc41-e7fcc59d5854' 2024-04-01T05:43:24.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:43:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8SQpm62fhOBAAqRU5qlpAYJTfFN91Rw2BMQ== 2024-04-01T05:43:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8SQpm62fhOBAAqRU5qlpAYJTfFN91Rw2BMQ=="}' 2024-04-01T05:43:24.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ca72484-20ba-49bb-bc41-e7fcc59d5854 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:43:25.177 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:43:25.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:43:25.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8SQpm62fhOBAAqRU5qlpAYJTfFN91Rw2BMQ== --osd-uuid 3ca72484-20ba-49bb-bc41-e7fcc59d5854 2024-04-01T05:43:25.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.216+0000 7fe3fec17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:25.227 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.228+0000 7fe3fec17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:25.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.229+0000 7fe3fec17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:25.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.229+0000 7fe3fec17640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:43:25.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.229+0000 7fe3fec17640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:43:25.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.229+0000 7fe3fec17640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:43:25.336 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:25.336+0000 7fe3fec17640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:43:30.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:43:30.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:43:30.877 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:43:30.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:43:30.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:43:31.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:43:31.253 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:43:31.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:43:31.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:43:31.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:43:31.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:43:31.278 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:31.279+0000 7fc664ad7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:31.304 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:31.304+0000 7fc664ad7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:31.305 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:31.305+0000 7fc664ad7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:31.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:43:31.537 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:43:31.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:31.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:31.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:31.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:31.832+0000 7fc664ad7640 -1 Falling back to public interface 2024-04-01T05:43:32.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:32.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:32.818 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:43:32.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:43:32.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:32.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:33.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:34.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:34.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:34.116 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:43:34.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:43:34.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:34.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:34.187 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:34.187+0000 7fc664ad7640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:43:34.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:35.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:35.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:35.422 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:43:35.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:43:35.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:35.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:35.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:36.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:36.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:36.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:43:36.743 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:43:36.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:36.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:37.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:38.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:38.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:38.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:43:38.034 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:43:38.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:38.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:38.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:39.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:39.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:39.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:43:39.330 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:43:39.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:39.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:39.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:40.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:40.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:40.630 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:43:40.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:43:40.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:40.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:40.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:41.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:41.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:41.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:43:41.931 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:43:41.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:41.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:42.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:43.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:43.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:43.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:43:43.226 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:43:43.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:43.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:43.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:44.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:44.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:44.519 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:43:44.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:43:44.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:44.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:44.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:45.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:45.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:45.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:43:45.930 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:43:45.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:45.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:46.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:47.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:47.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:47.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:43:47.462 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:43:47.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:47.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:48.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:49.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:49.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:49.158 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:43:49.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:43:49.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:49.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:49.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:50.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:50.854 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:43:50.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:50.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:43:50.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:50.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:51.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:52.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:52.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:52.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:43:52.415 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:43:52.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:52.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:53.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:54.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:43:54.151 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:43:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:54.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:54.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:55.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:55.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:55.845 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:43:55.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:43:55.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:55.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:56.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:43:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:43:57.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:43:57.307 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:43:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:43:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:43:57.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:43:57.594 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3724834296,v1:127.0.0.1:6803/3724834296] [v2:127.0.0.1:6804/3724834296,v1:127.0.0.1:6805/3724834296] exists,up 3ca72484-20ba-49bb-bc41-e7fcc59d5854 2024-04-01T05:43:57.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:43:57.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:43:57.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:43:57.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-04-01T05:43:57.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:43:57.599 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:43:57.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:43:57.601 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:43:57.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 13abe988-7381-475d-b1ac-846aae1c5d42' 2024-04-01T05:43:57.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:43:57.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdSQpmN7zEJBAAIgjJzY0mkREwgYAnve4/pg== 2024-04-01T05:43:57.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdSQpmN7zEJBAAIgjJzY0mkREwgYAnve4/pg=="}' 2024-04-01T05:43:57.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13abe988-7381-475d-b1ac-846aae1c5d42 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:43:57.949 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:43:57.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:43:57.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdSQpmN7zEJBAAIgjJzY0mkREwgYAnve4/pg== --osd-uuid 13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:43:57.989 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:57.988+0000 7fcbddc9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:58.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:58.025+0000 7fcbddc9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:58.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:58.026+0000 7fcbddc9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:43:58.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:58.026+0000 7fcbddc9d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:43:58.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:58.026+0000 7fcbddc9d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:43:58.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:58.026+0000 7fcbddc9d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:43:58.092 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:43:58.092+0000 7fcbddc9d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:44:03.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:44:03.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:44:03.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:44:03.865 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:44:03.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:44:04.234 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:44:04.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:44:04.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:44:04.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:44:04.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:44:04.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:44:04.261 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:04.260+0000 7f4fab089640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:04.262 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:04.262+0000 7f4fab089640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:04.263 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:04.263+0000 7f4fab089640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:04.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:44:04.563 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:44:04.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:04.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:04.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:04.777+0000 7f4fab089640 -1 Falling back to public interface 2024-04-01T05:44:04.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:05.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:05.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:05.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:44:05.850 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:44:05.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:05.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:06.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:07.099 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:07.099+0000 7f4fab089640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:44:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:07.145 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:44:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:44:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:07.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:07.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:08.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:08.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:08.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:44:08.560 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:44:08.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:08.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:08.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:09.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:09.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:09.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:44:09.884 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:44:09.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:09.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:10.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:11.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:11.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:11.297 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:44:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:44:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:11.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:12.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:12.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:12.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:44:12.592 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:44:12.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:12.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:12.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:13.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:13.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:13.899 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:44:13.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:44:13.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:13.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:14.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:15.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:15.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:15.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:44:15.264 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:44:15.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:15.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:15.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:16.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:16.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:16.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:44:16.560 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:44:16.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:16.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:16.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:17.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:17.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:17.853 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:44:17.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:44:17.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:17.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:18.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:19.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:19.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:19.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:44:19.147 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:44:19.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:19.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:19.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:20.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:44:20.791 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:44:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:20.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:21.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:22.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:22.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:22.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:44:22.201 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:44:22.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:22.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:22.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:23.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:23.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:23.841 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:44:23.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:44:23.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:23.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:24.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:25.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:25.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:25.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:44:25.192 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:44:25.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:25.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:25.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:44:26.491 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:44:26.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:26.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:27.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:28.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:28.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:28.007 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:44:28.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:44:28.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:28.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:28.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:29.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:29.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:29.301 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:44:29.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:44:29.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:29.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:29.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:30.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:30.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:30.697 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:44:30.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:44:30.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:30.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:30.989 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1015976536,v1:127.0.0.1:6811/1015976536] [v2:127.0.0.1:6812/1015976536,v1:127.0.0.1:6813/1015976536] exists,up 13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:44:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:44:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:44:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:44:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-04-01T05:44:30.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:44:31.262 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:44:31.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:44:31.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:44:31.677 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:44:31.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:44:32.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:44:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-04-01T05:44:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:44:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:44:34.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:44:34.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:44:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:44:34.338 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:44:34.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:44:34.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:44:34.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:44:34.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:44:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:44:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:44:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:44:34.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:44:34.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:44:34.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:44:34.719 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:44:34.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:44:34.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:44:34.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:44:34.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-04-01T05:44:34.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-04-01T05:44:34.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-04-01T05:44:34.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:44:34.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:44:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-04-01T05:44:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-04-01T05:44:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672964' 2024-04-01T05:44:34.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:44:34.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-04-01T05:44:34.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:44:34.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:44:34.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-04-01T05:44:34.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:44:34.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-04-01T05:44:34.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-04-01T05:44:34.932 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836491 2024-04-01T05:44:34.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:44:35.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-04-01T05:44:35.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:44:35.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-04-01T05:44:35.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:44:35.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:44:35.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-04-01T05:44:35.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:44:35.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-04-01T05:44:35.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-04-01T05:44:35.241 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672964 2024-04-01T05:44:35.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:44:35.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-04-01T05:44:35.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:44:35.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:44:35.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:44:35.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:44:35.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:44:35.880 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:44:35.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:44:35.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:44:35.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:44:35.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:44:35.881 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:44:36.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:44:36.202 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:44:36.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:44:36.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:44:36.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:44:36.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:44:36.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:44:36.930 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:44:36.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:44:37.356 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:44:37.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:44:37.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:44:37.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-04-01T05:44:37.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-04-01T05:44:37.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-04-01T05:44:37.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-04-01T05:44:37.620 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-04-01T05:44:37.620 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-04-01T05:44:37.620 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:44:37.621 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-04-01T05:44:37.621 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:44:37.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T05:44:37.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-04-01T05:44:37.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:44:38.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:44:38.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:44:38.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-04-01T05:44:40.087 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:44:40.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:44:40.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:44:40.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:44:40.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:44:40.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:44:40.635 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:40.635+0000 7fa8d2211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:40.637 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:40.637+0000 7fa8d2211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:40.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:40.638+0000 7fa8d2211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:40.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:44:41.159 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:41.160+0000 7fa8d2211640 -1 Falling back to public interface 2024-04-01T05:44:41.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:44:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:44:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:42.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:44:42.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:43.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:43.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:43.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:44:43.506 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:44:43.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:43.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:44:43.578 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:43.577+0000 7fa8d2211640 -1 osd.0 20 log_to_monitors true 2024-04-01T05:44:43.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:44:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:44:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/3527202807,v1:127.0.0.1:6803/3527202807] [v2:127.0.0.1:6804/3527202807,v1:127.0.0.1:6805/3527202807] exists,up 3ca72484-20ba-49bb-bc41-e7fcc59d5854 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:44:45.100 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:44:45.101 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:44:45.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:44:45.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:44:45.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:44:45.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:44:45.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:44:45.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:44:45.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:44:45.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:44:45.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:44:45.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:44:45.476 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:44:45.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:44:45.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:44:45.476 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:44:45.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-04-01T05:44:45.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-04-01T05:44:45.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-04-01T05:44:45.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:44:45.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:44:45.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-04-01T05:44:45.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-04-01T05:44:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2024-04-01T05:44:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:44:45.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-04-01T05:44:45.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:44:45.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:44:45.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-04-01T05:44:45.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:44:45.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-04-01T05:44:45.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-04-01T05:44:45.690 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182402 2024-04-01T05:44:45.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:44:45.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-04-01T05:44:45.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:44:46.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:44:46.974 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:44:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-04-01T05:44:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:44:47.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-04-01T05:44:47.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:44:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:44:47.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-04-01T05:44:47.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:44:47.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-04-01T05:44:47.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-04-01T05:44:47.312 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672968 2024-04-01T05:44:47.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:44:47.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-04-01T05:44:47.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:44:47.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:44:47.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:44:47.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:44:47.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:44:48.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:44:48.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:44:48.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:44:48.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:44:48.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:44:48.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:44:48.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:44:48.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-04-01T05:44:48.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-04-01T05:44:48.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:44:48.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-04-01T05:44:48.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:44:48.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:44:48.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:44:49.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:44:31.617145+0000 2024-04-01T05:44:49.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-04-01T05:44:49.420 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 1.3 on osd.1 to repair 2024-04-01T05:44:49.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-04-01T05:44:31.617145+0000 2024-04-01T05:44:49.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-04-01T05:44:49.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:44:31.617145+0000 2024-04-01T05:44:49.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:44:49.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:44:49.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:44:49.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:44:49.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:44:49.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:44:49.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:44:49.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:44:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:31.617145+0000 '>' 2024-04-01T05:44:31.617145+0000 2024-04-01T05:44:49.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:44:50.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:44:50.712 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:44:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:44:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:44:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:44:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:44:50.712 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:44:50.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:31.617145+0000 '>' 2024-04-01T05:44:31.617145+0000 2024-04-01T05:44:50.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:44:52.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:44:52.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:44:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:44:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:44:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:44:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:44:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:50.200094+0000 '>' 2024-04-01T05:44:31.617145+0000 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:44:52.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:44:52.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:44:52.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:44:52.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:44:52.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:44:52.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:44:52.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:44:52.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:44:52.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:44:52.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:44:52.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:44:52.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:44:52.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:44:52.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:44:53.907 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:44:53.908 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:44:53.908 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:44:54.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:44:54.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:44:54.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:44:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:44:54.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:44:54.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:44:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:44:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:44:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:44:54.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:44:54.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:54.204+0000 7fa07be0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:54.206 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:54.207+0000 7fa07be0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:54.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:54.208+0000 7fa07be0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:54.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:54.725 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:54.725+0000 7fa07be0d640 -1 Falling back to public interface 2024-04-01T05:44:54.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:55.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:55.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:55.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:44:55.759 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:44:55.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:55.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:56.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:44:56.904 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:44:56.904+0000 7fa07be0d640 -1 osd.1 26 log_to_monitors true 2024-04-01T05:44:57.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:44:57.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:44:57.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:44:57.053 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:44:57.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:44:57.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 30 up_thru 25 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1278961023,v1:127.0.0.1:6811/1278961023] [v2:127.0.0.1:6812/1278961023,v1:127.0.0.1:6813/1278961023] exists,up 13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:44:57.518 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:44:57.519 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:44:57.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:44:57.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:44:57.519 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:44:57.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:44:57.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:44:57.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:44:57.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:44:57.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:44:57.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:44:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:44:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:44:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:44:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:44:57.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:44:58.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-04-01T05:44:58.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-04-01T05:44:58.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-04-01T05:44:58.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:44:58.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:44:58.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-04-01T05:44:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-04-01T05:44:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-04-01T05:44:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:44:58.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-04-01T05:44:58.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:44:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:44:58.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-04-01T05:44:58.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:44:58.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-04-01T05:44:58.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-04-01T05:44:58.333 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182405 2024-04-01T05:44:58.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:44:58.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-04-01T05:44:58.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:44:59.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:44:59.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:44:59.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182405 2024-04-01T05:44:59.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:44:59.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-04-01T05:44:59.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:44:59.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:44:59.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-04-01T05:44:59.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:44:59.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-04-01T05:44:59.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-04-01T05:44:59.915 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 128849018882 2024-04-01T05:44:59.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:45:00.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-04-01T05:45:00.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:45:00.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:45:00.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:45:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:45:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:45:00.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:45:00.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:45:00.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:45:00.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:45:00.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:45:00.551 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:45:00.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:45:00.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:45:00.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:45:00.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:45:01.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:45:01.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:45:01.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:45:01.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:45:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:45:01.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-04-01T05:45:01.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-04-01T05:45:01.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-04-01T05:45:01.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-04-01T05:45:01.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:45:01.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:45:01.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:45:01.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:45:01.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:45:01.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:45:01.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:45:01.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:45:01.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:45:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-04-01T05:45:02.947 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-04-01T05:45:03.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:45:03.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:45:03.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:45:03.470 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:45:03.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:45:03.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:45:03.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:45:03.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:45:03.472 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:45:03.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:45:03.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:45:03.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:45:03.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:45:03.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:45:03.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:45:03.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:03.499+0000 7f806d611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:03.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:03.501+0000 7f806d611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:03.502 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:03.502+0000 7f806d611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:03.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:04.024 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:04.024+0000 7f806d611640 -1 Falling back to public interface 2024-04-01T05:45:04.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:05.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:05.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:05.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:45:05.080 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:45:05.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:05.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:05.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:06.372+0000 7f806d611640 -1 osd.1 31 log_to_monitors true 2024-04-01T05:45:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:45:06.376 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:45:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:06.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:06.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:07.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:07.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:07.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:45:07.690 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:45:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:07.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/2928290438,v1:127.0.0.1:6811/2928290438] [v2:127.0.0.1:6812/2928290438,v1:127.0.0.1:6813/2928290438] exists,up 13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:45:07.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:45:07.986 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:45:07.986 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:45:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:45:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:45:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:45:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:45:08.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:45:08.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:45:08.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:45:08.351 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:45:08.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:45:08.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:45:08.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:45:08.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182408 2024-04-01T05:45:08.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182408 2024-04-01T05:45:08.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182408' 2024-04-01T05:45:08.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:45:08.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:45:08.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-04-01T05:45:08.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-04-01T05:45:08.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182408 1-150323855362' 2024-04-01T05:45:08.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:45:08.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182408 2024-04-01T05:45:08.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:45:08.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:45:08.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182408 2024-04-01T05:45:08.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:45:08.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182408 2024-04-01T05:45:08.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182408' 2024-04-01T05:45:08.565 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182408 2024-04-01T05:45:08.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:45:08.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182408 2024-04-01T05:45:08.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:45:09.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:45:09.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:45:10.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182408 2024-04-01T05:45:10.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:45:10.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-04-01T05:45:10.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:45:10.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:45:10.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-04-01T05:45:10.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:45:10.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-04-01T05:45:10.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-04-01T05:45:10.141 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 150323855362 2024-04-01T05:45:10.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:45:10.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-04-01T05:45:10.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:45:10.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:45:10.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:45:10.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:45:10.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:45:11.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:45:11.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:45:11.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:45:11.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:45:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:45:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:45:11.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:45:11.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-04-01T05:45:11.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-04-01T05:45:11.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-04-01T05:45:11.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-04-01T05:45:11.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:45:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-04-01T05:45:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-04-01T05:45:11.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-04-01T05:45:11.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-04-01T05:45:11.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:45:11.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:45:11.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:45:11.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:45:11.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:11.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-04-01T05:45:12.276 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 1.3 on osd.1 to repair 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:45:12.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:45:12.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:50.200094+0000 '>' 2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:12.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:45:13.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:45:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:45:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:45:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:45:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:45:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:45:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:45:13.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:50.200094+0000 '>' 2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:13.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:45:14.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:45:14.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:45:14.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:45:14.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:45:14.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:45:14.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:45:14.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:45:15.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:50.200094+0000 '>' 2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:15.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:45:16.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:45:16.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:45:16.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:45:16.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:45:16.156 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:45:16.157 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:45:16.157 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:45:16.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:44:50.200094+0000 '>' 2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:16.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:45:17.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:45:17.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:45:17.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-04-01T05:45:17.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-04-01T05:45:17.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:45:17.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:45:17.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-04-01T05:45:17.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:45:12.475603+0000 '>' 2024-04-01T05:44:50.200094+0000 2024-04-01T05:45:17.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:45:17.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:45:17.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:45:17.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:45:17.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:45:17.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:45:17.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:45:18.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:45:18.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-04-01T05:45:18.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:45:18.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:45:18.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:45:18.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:45:18.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:45:18.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-04-01T05:45:19.096 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-04-01T05:45:19.096 INFO:tasks.workunit.client.0.smithi067.stdout:_ 2024-04-01T05:45:19.096 INFO:tasks.workunit.client.0.smithi067.stdout:snapset 2024-04-01T05:45:19.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:45:19.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:45:19.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:45:19.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:45:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:45:19.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:45:19.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:45:19.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:45:19.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:45:19.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:45:19.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:45:19.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:45:19.398 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:19.398+0000 7fb81ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:19.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:19.400+0000 7fb81ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:19.401 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:19.401+0000 7fb81ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:19.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:19.919 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:19.920+0000 7fb81ca0d640 -1 Falling back to public interface 2024-04-01T05:45:19.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:20.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:20.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:20.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:45:20.947 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:45:20.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:20.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:21.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:22.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:22.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:22.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:45:22.243 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:45:22.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:22.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:22.327 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:22.327+0000 7fb81ca0d640 -1 osd.1 37 log_to_monitors true 2024-04-01T05:45:22.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:23.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:23.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:23.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:45:23.532 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:45:23.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:23.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:45:23.867 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/1629602384,v1:127.0.0.1:6811/1629602384] [v2:127.0.0.1:6812/1629602384,v1:127.0.0.1:6813/1629602384] exists,up 13abe988-7381-475d-b1ac-846aae1c5d42 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:45:23.868 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:45:23.869 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:45:23.869 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:45:23.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:45:23.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:45:23.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:45:23.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:45:23.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:45:23.953 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:45:24.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:45:24.234 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:45:24.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:45:24.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:45:24.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:45:24.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182413 2024-04-01T05:45:24.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182413 2024-04-01T05:45:24.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182413' 2024-04-01T05:45:24.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:45:24.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:45:24.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-04-01T05:45:24.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-04-01T05:45:24.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182413 1-176093659138' 2024-04-01T05:45:24.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:45:24.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182413 2024-04-01T05:45:24.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:45:24.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:45:24.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182413 2024-04-01T05:45:24.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:45:24.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182413 2024-04-01T05:45:24.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182413' 2024-04-01T05:45:24.447 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 107374182413 2024-04-01T05:45:24.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:45:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182411 -lt 107374182413 2024-04-01T05:45:24.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:45:25.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:45:25.729 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:45:26.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182413 -lt 107374182413 2024-04-01T05:45:26.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:45:26.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659138 2024-04-01T05:45:26.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:45:26.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:45:26.037 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659138 2024-04-01T05:45:26.037 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:45:26.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-04-01T05:45:26.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-04-01T05:45:26.038 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 176093659138 2024-04-01T05:45:26.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:45:26.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-04-01T05:45:26.326 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:45:26.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:45:26.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:45:26.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:45:26.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:45:26.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:45:26.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:45:26.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:45:26.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:45:26.680 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:45:26.680 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:45:26.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:45:26.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:45:26.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:45:26.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:45:27.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:45:27.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:45:27.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:45:27.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-04-01T05:45:27.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:45:27.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:45:27.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:45:27.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:45:27.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:45:27.467 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:45:27.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:45:27.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:45:27.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:45:27.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:45:27.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:45:27.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:27.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:27.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:45:27.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:45:27.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:45:27.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:45:27.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:45:27.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:45:27.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:45:27.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:45:27.485 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:45:27.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:45:27.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:45:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:45:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:45:27.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:45:27.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:45:27.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:45:27.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:45:27.488 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:45:27.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:45:27.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:45:27.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:45:27.491 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:45:27.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:27.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:27.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:45:27.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:45:27.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:45:27.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:45:27.493 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:45:27.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:27.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:27.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-04-01T05:45:27.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:45:27.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:45:27.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:27.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:45:28.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:45:28.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:45:28.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:45:28.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:45:28.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:45:28.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:45:28.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:45:28.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:45:28.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:45:28.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:45:28.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:45:28.141 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:45:28.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:45:28.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:45:28.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:45:29.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:45:29.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:45:29.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:45:29.040 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:45:29.040 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:29.040 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:29.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:29.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:45:29.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:45:29.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-04-01T05:45:29.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-04-01T05:45:29.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:45:29.075 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:45:29.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:45:29.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a' 2024-04-01T05:45:29.078 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:45:29.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:45:29.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5SQpmBrGWBRAAXPRs4ARM7vyB1IWr1wnTPw== 2024-04-01T05:45:29.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5SQpmBrGWBRAAXPRs4ARM7vyB1IWr1wnTPw=="}' 2024-04-01T05:45:29.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a -i td/osd-scrub-repair/0/new.json 2024-04-01T05:45:29.302 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:45:29.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:45:29.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5SQpmBrGWBRAAXPRs4ARM7vyB1IWr1wnTPw== --osd-uuid ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:45:29.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.340+0000 7fa3f2811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:29.353 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.352+0000 7fa3f2811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:29.354 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.353+0000 7fa3f2811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:29.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.354+0000 7fa3f2811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:45:29.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.354+0000 7fa3f2811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:45:29.355 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.354+0000 7fa3f2811640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:45:29.445 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:29.444+0000 7fa3f2811640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:45:35.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:45:35.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:45:35.152 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:45:35.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:45:35.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:45:35.522 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:45:35.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:45:35.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:45:35.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:45:35.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:45:35.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:45:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:35.547+0000 7f02f3b91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:35.549 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:35.549+0000 7f02f3b91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:35.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:35.550+0000 7f02f3b91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:35.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:45:35.817 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:45:35.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:35.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:36.077 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:36.077+0000 7f02f3b91640 -1 Falling back to public interface 2024-04-01T05:45:36.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:37.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:37.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:37.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:45:37.099 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:45:37.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:37.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:37.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:38.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:38.398 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:45:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:45:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:38.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:38.403 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:45:38.403+0000 7f02f3b91640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:45:38.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:39.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:39.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:39.694 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:45:39.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:45:39.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:39.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:39.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:40.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:40.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:40.988 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:45:40.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:45:40.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:40.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:41.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:42.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:42.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:42.374 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:45:42.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:45:42.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:42.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:42.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:43.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:43.669 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:45:43.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:45:43.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:43.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:43.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:44.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:44.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:44.966 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:45:44.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:45:44.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:44.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:45.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:46.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:46.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:46.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:45:46.343 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:45:46.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:46.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:46.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:47.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:47.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:47.637 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:45:47.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:45:47.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:47.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:48.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:49.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:49.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:49.005 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:45:49.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:45:49.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:49.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:49.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:50.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:50.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:50.347 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:45:50.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:45:50.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:50.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:50.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:51.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:51.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:51.907 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:45:51.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:45:51.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:51.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:52.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:53.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:53.724 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:45:53.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:53.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:45:53.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:53.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:54.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:55.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:55.270 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:45:55.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:55.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:45:55.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:55.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:56.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:57.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:57.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:57.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:45:57.040 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:45:57.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:57.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:57.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:45:58.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:45:58.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:45:58.783 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:45:58.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:45:58.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:45:58.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:45:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:00.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:46:00.371 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:46:00.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:00.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:46:00.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:01.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:01.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:01.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:46:01.765 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:46:01.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:01.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:46:02.056 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1146222861,v1:127.0.0.1:6803/1146222861] [v2:127.0.0.1:6804/1146222861,v1:127.0.0.1:6805/1146222861] exists,up ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:46:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:46:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:46:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:46:02.064 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:46:02.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:46:02.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62979ae6-1ee9-4705-8b55-9b221f3e1caf' 2024-04-01T05:46:02.064 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:46:02.064 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:46:02.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaSgpmDVGnBBAAh4+6Wv8P9OvtGolOPdktbQ== 2024-04-01T05:46:02.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaSgpmDVGnBBAAh4+6Wv8P9OvtGolOPdktbQ=="}' 2024-04-01T05:46:02.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62979ae6-1ee9-4705-8b55-9b221f3e1caf -i td/osd-scrub-repair/1/new.json 2024-04-01T05:46:02.455 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:46:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:46:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaSgpmDVGnBBAAh4+6Wv8P9OvtGolOPdktbQ== --osd-uuid 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:46:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.495+0000 7f17dea13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:02.531 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.532+0000 7f17dea13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:02.532 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.533+0000 7f17dea13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.533+0000 7f17dea13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:46:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.533+0000 7f17dea13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:46:02.533 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.533+0000 7f17dea13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:46:02.609 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:02.609+0000 7f17dea13640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:46:08.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:46:08.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:46:08.641 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:46:08.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:46:08.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:46:09.015 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:46:09.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:46:09.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:46:09.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:46:09.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:46:09.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:46:09.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:09.042+0000 7f48857ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:09.043 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:09.044+0000 7f48857ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:09.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:09.045+0000 7f48857ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:09.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:46:09.309 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:46:09.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:09.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:09.558 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:09.558+0000 7f48857ac640 -1 Falling back to public interface 2024-04-01T05:46:09.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:10.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:10.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:10.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:46:10.591 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:46:10.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:10.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:10.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:11.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:11.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:11.879 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:46:11.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:46:11.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:11.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:11.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:11.931+0000 7f48857ac640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:46:12.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:13.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:13.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:13.169 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:46:13.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:46:13.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:13.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:13.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:14.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:14.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:14.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:46:14.463 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:46:14.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:14.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:14.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:15.787 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:46:15.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:46:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:15.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:16.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:17.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:17.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:17.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:46:17.081 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:46:17.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:17.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:17.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:18.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:46:18.377 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:46:18.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:18.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:18.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:19.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:19.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:46:19.788 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:46:19.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:19.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:20.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:21.082 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:46:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:46:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:21.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:22.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:22.376 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:46:22.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:22.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:46:22.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:22.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:22.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:23.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:23.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:23.672 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:46:23.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:46:23.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:23.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:24.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:25.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:25.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:25.260 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:46:25.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:46:25.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:25.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:25.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:26.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:26.738 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:46:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:46:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:26.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:27.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:28.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:28.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:28.445 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:46:28.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:46:28.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:28.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:29.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:30.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:30.080 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:46:30.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:30.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:46:30.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:30.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:30.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:31.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:31.867 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:46:31.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:31.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:46:31.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:31.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:32.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:33.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:46:33.428 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:46:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:33.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:33.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:34.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:34.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:34.839 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:46:34.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:46:34.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:34.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:46:35.132 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1763851898,v1:127.0.0.1:6811/1763851898] [v2:127.0.0.1:6812/1763851898,v1:127.0.0.1:6813/1763851898] exists,up 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:46:35.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:46:35.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:46:35.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:46:35.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 2024-04-01T05:46:35.139 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 2024-04-01T05:46:35.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51' 2024-04-01T05:46:35.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:46:35.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7SgpmZ58tCRAAJqYl8hielGG3b5Hm3R2Lvg== 2024-04-01T05:46:35.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7SgpmZ58tCRAAJqYl8hielGG3b5Hm3R2Lvg=="}' 2024-04-01T05:46:35.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:46:35.551 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:46:35.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:46:35.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7SgpmZ58tCRAAJqYl8hielGG3b5Hm3R2Lvg== --osd-uuid 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 2024-04-01T05:46:35.590 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.590+0000 7efc58a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:35.627 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.627+0000 7efc58a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:35.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.628+0000 7efc58a0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:35.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.628+0000 7efc58a0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:46:35.628 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.628+0000 7efc58a0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:46:35.629 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.628+0000 7efc58a0e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:46:35.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:35.703+0000 7efc58a0e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:46:41.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:46:41.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:46:41.667 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:46:41.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:46:41.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:46:42.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:46:42.037 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:46:42.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:46:42.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:46:42.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:46:42.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:46:42.063 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:42.063+0000 7fce8a812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:42.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:42.065+0000 7fce8a812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:42.066 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:42.066+0000 7fce8a812640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:42.330 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:46:42.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:46:42.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:42.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:42.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:42.600+0000 7fce8a812640 -1 Falling back to public interface 2024-04-01T05:46:42.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:43.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:43.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:43.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:46:43.612 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:46:43.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:43.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:43.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:44.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:44.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:44.901 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:46:44.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:46:44.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:44.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:44.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:46:44.934+0000 7fce8a812640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:46:45.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:46.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:46.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:46.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:46:46.295 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:46:46.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:46.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:46.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:47.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:47.625 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:46:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:46:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:47.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:48.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:49.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:49.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:49.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:46:49.012 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:46:49.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:49.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:49.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:50.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:50.330 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:46:50.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:50.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:46:50.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:50.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:50.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:51.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:51.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:51.647 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:46:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:46:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:52.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:53.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:53.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:53.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:46:53.034 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:46:53.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:53.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:53.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:54.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:54.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:54.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:46:54.334 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:46:54.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:54.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:54.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:55.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:55.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:55.654 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:46:55.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:46:55.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:55.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:55.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:56.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:56.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:56.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:46:56.985 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:46:56.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:56.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:57.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:58.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:58.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:58.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:46:58.281 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:46:58.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:58.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:46:58.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:46:59.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:46:59.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:46:59.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:46:59.808 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:46:59.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:46:59.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:00.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:01.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:01.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:01.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:47:01.629 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:47:01.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:01.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:01.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:02.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:02.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:02.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:47:02.923 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:47:02.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:02.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:03.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:04.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:04.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:04.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:47:04.244 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:47:04.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:04.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:04.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:05.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:05.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:05.864 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:47:05.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:47:05.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:05.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:06.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:07.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:07.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:07.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:47:07.428 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:47:07.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:07.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:07.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:08.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:08.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:08.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T05:47:08.794 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T05:47:08.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:08.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:47:09.091 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1680366084,v1:127.0.0.1:6819/1680366084] [v2:127.0.0.1:6820/1680366084,v1:127.0.0.1:6821/1680366084] exists,up 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 2024-04-01T05:47:09.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:47:09.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:47:09.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:47:09.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-04-01T05:47:09.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:47:09.407 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:47:09.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:47:09.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:47:09.792 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:47:09.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:47:10.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:47:12.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-04-01T05:47:12.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-04-01T05:47:13.389 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' created 2024-04-01T05:47:13.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:47:14.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-04-01T05:47:14.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:47:14.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:47:14.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-04-01T05:47:14.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:47:14.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-04-01T05:47:14.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:47:14.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:47:15.518 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:47:15.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:47:16.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-04-01T05:47:16.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:47:16.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:47:16.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:47:16.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:47:16.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:47:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:47:16.533 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:47:16.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:47:16.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:47:16.533 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:47:16.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:47:16.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:47:16.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:47:16.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:47:16.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:47:16.630 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:47:16.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:47:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:47:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:47:16.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:16.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:47:17.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-04-01T05:47:17.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-04-01T05:47:17.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-04-01T05:47:17.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:17.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:47:17.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-04-01T05:47:17.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-04-01T05:47:17.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672980' 2024-04-01T05:47:17.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:17.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:47:17.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-04-01T05:47:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-04-01T05:47:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672980 2-64424509449' 2024-04-01T05:47:17.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:17.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-04-01T05:47:17.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:17.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:47:17.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-04-01T05:47:17.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:17.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-04-01T05:47:17.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-01T05:47:17.416 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836500 2024-04-01T05:47:17.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:17.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-04-01T05:47:17.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:17.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-04-01T05:47:17.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:17.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:47:17.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-04-01T05:47:17.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:17.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-04-01T05:47:17.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-04-01T05:47:17.729 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672980 2024-04-01T05:47:17.729 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-04-01T05:47:18.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:47:19.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:47:19.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-04-01T05:47:19.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:47:20.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:47:20.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:20.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-04-01T05:47:20.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:20.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-04-01T05:47:20.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:20.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:47:20.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-04-01T05:47:20.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:20.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-04-01T05:47:20.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-01T05:47:20.671 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509449 2024-04-01T05:47:20.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:47:20.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-01T05:47:20.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:47:20.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:20.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:21.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:47:21.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:47:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:47:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:47:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:47:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:47:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:47:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:47:21.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:47:21.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:47:21.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:21.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:21.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:47:21.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:47:22.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:47:22.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:47:22.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:47:22.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:47:22.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:47:22.040 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:47:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:47:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:47:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:47:22.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:22.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:47:22.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-04-01T05:47:22.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-04-01T05:47:22.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-04-01T05:47:22.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:22.427 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:47:22.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-04-01T05:47:22.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-04-01T05:47:22.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672982' 2024-04-01T05:47:22.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:22.531 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:47:22.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-04-01T05:47:22.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-04-01T05:47:22.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672982 2-64424509451' 2024-04-01T05:47:22.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:22.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-04-01T05:47:22.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:22.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:47:22.638 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-04-01T05:47:22.638 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:22.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-04-01T05:47:22.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-01T05:47:22.640 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836502 2024-04-01T05:47:22.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:22.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836502 2024-04-01T05:47:22.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:47:23.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:47:23.924 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:24.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-01T05:47:24.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:24.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-04-01T05:47:24.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:24.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:47:24.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-04-01T05:47:24.240 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:24.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-04-01T05:47:24.241 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672982 2024-04-01T05:47:24.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-01T05:47:24.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-01T05:47:24.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:24.524 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-04-01T05:47:24.525 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:24.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:47:24.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-04-01T05:47:24.526 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:24.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-04-01T05:47:24.527 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509451 2024-04-01T05:47:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-01T05:47:24.528 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:47:24.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-01T05:47:24.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:47:24.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:24.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:47:25.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:47:25.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:47:25.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:47:25.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:47:25.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:25.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:25.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:47:25.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:47:25.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:47:25.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:47:25.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:47:26.178 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:47:26.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:47:26.748 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:47:26.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:47:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:47:26.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:47:27.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:27.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:47:27.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:47:27.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:47:29.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:47:29.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:47:29.429 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:47:29.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:47:29.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:47:29.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:47:29.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:47:29.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:29.455+0000 7f1f0c16f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:29.457 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:29.458+0000 7f1f0c16f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:29.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:29.459+0000 7f1f0c16f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:29.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:29.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:29.975+0000 7f1f0c16f640 -1 Falling back to public interface 2024-04-01T05:47:30.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:31.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:31.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:31.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:47:31.008 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:31.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:31.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:31.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:32.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:47:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:47:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:32.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:32.400 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:32.400+0000 7f1f0c16f640 -1 osd.1 36 log_to_monitors true 2024-04-01T05:47:32.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:33.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:33.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:33.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:47:33.622 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:47:33.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:33.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1731144698,v1:127.0.0.1:6811/1731144698] [v2:127.0.0.1:6812/1731144698,v1:127.0.0.1:6813/1731144698] exists,up 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:47:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:47:33.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:47:33.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:47:33.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:47:34.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:47:34.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:47:34.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:47:34.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:47:34.007 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:47:34.007 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:47:34.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:47:34.359 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:34.359 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:47:34.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:47:34.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:34.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:47:34.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-04-01T05:47:34.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-04-01T05:47:34.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-04-01T05:47:34.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:34.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:47:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-04-01T05:47:34.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-04-01T05:47:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-171798691842' 2024-04-01T05:47:34.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:34.596 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:47:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-04-01T05:47:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-04-01T05:47:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-171798691842 2-64424509455' 2024-04-01T05:47:34.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:34.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-04-01T05:47:34.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:34.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:47:34.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-04-01T05:47:34.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:34.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-04-01T05:47:34.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-01T05:47:34.704 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836505 2024-04-01T05:47:34.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-04-01T05:47:34.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:47:35.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:47:35.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:36.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-01T05:47:36.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:36.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-04-01T05:47:36.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:36.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:47:36.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-04-01T05:47:36.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:36.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-04-01T05:47:36.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-04-01T05:47:36.287 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 171798691842 2024-04-01T05:47:36.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:36.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-04-01T05:47:36.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:36.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-04-01T05:47:36.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:36.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:47:36.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-04-01T05:47:36.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:36.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-04-01T05:47:36.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-01T05:47:36.594 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509455 2024-04-01T05:47:36.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:47:36.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-01T05:47:36.878 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:47:36.878 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:36.878 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:37.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:47:37.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:47:37.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:47:37.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:47:37.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:47:37.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:47:37.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:47:37.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:47:37.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:47:37.516 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:47:37.516 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:37.516 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:37.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:47:37.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:47:38.281 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:47:38.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:47:38.808 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:47:38.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:47:38.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:47:38.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:47:39.022 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-04-01T05:47:39.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-04-01T05:47:39.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:47:39.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-04-01T05:47:39.025 INFO:tasks.workunit.client.0.smithi067.stderr:1+0 records in 2024-04-01T05:47:39.025 INFO:tasks.workunit.client.0.smithi067.stderr:1+0 records out 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.2274e-05 s, 32.9 MB/s 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:47:39.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:47:39.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:47:39.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:47:39.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:39.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:47:39.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:47:39.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:47:39.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:47:39.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:47:39.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:47:41.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:47:41.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:47:41.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:47:41.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:47:41.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:47:41.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:47:41.503 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:41.502+0000 7fca0cc97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:41.505 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:41.504+0000 7fca0cc97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:41.506 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:41.505+0000 7fca0cc97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:41.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:47:42.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:42.025+0000 7fca0cc97640 -1 Falling back to public interface 2024-04-01T05:47:42.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:43.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:43.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:43.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:47:43.062 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:43.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:43.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:47:43.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:44.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:44.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:44.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:47:44.356 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:47:44.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:44.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:47:44.489 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:44.488+0000 7fca0cc97640 -1 osd.0 44 log_to_monitors true 2024-04-01T05:47:44.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:45.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:45.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:45.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:47:45.701 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:47:45.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:45.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/3452465765,v1:127.0.0.1:6803/3452465765] [v2:127.0.0.1:6804/3452465765,v1:127.0.0.1:6805/3452465765] exists,up ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:47:46.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:47:46.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:47:46.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:47:46.100 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:47:46.100 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:47:46.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:47:46.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:47:46.101 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:47:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:47:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:47:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:47:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:47:46.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:47:46.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:47:46.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:47:46.469 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:46.469 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:47:46.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:47:46.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:46.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:47:46.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-04-01T05:47:46.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-04-01T05:47:46.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506' 2024-04-01T05:47:46.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:46.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:47:46.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-04-01T05:47:46.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-04-01T05:47:46.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845' 2024-04-01T05:47:46.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:46.680 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:47:46.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-04-01T05:47:46.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-04-01T05:47:46.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845 2-64424509458' 2024-04-01T05:47:46.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:46.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397506 2024-04-01T05:47:46.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:46.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:47:46.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397506 2024-04-01T05:47:46.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:46.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-04-01T05:47:46.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-04-01T05:47:46.789 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 210453397506 2024-04-01T05:47:46.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:47.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 210453397506 2024-04-01T05:47:47.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:47:48.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:47:48.074 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:48.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-04-01T05:47:48.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-04-01T05:47:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:48.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:47:48.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-04-01T05:47:48.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:48.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-04-01T05:47:48.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-04-01T05:47:48.376 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 171798691845 2024-04-01T05:47:48.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:48.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-04-01T05:47:48.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:48.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-04-01T05:47:48.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:48.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:47:48.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-04-01T05:47:48.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:48.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-04-01T05:47:48.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-01T05:47:48.660 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509458 2024-04-01T05:47:48.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:47:48.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-01T05:47:48.943 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:47:48.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:48.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:49.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:47:49.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:47:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:47:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:47:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:47:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:47:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:47:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:47:49.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:47:49.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:47:49.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:49.577 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:47:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:47:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:47:49.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:47:50.421 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:47:50.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:47:50.815 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:47:50.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:47:50.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:47:50.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:47:51.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-04-01T05:47:51.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:47:51.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:47:51.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:47:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:47:51.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-04-01T05:47:53.220 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:47:53.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:47:53.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:47:53.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:47:53.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:47:53.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:53.767+0000 7f221adac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:53.769 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:53.770+0000 7f221adac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:53.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:53.771+0000 7f221adac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:47:54.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:54.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:54.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:54.285+0000 7f221adac640 -1 Falling back to public interface 2024-04-01T05:47:54.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:55.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:55.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:55.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:47:55.318 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:55.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:55.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:55.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:56.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:56.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:56.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:47:56.613 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:47:56.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:56.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:56.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:47:56.661+0000 7f221adac640 -1 osd.1 53 log_to_monitors true 2024-04-01T05:47:57.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:47:58.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:47:58.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:47:58.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:47:58.032 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:47:58.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:47:58.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/3275421267,v1:127.0.0.1:6811/3275421267] [v2:127.0.0.1:6812/3275421267,v1:127.0.0.1:6813/3275421267] exists,up 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:47:58.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:47:58.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:47:58.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:47:58.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:47:58.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:47:58.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:47:58.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:47:58.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:47:58.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:47:58.787 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:47:58.787 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:47:58.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:47:58.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:58.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:47:58.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-04-01T05:47:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-04-01T05:47:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509' 2024-04-01T05:47:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:58.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:47:58.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-04-01T05:47:58.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-04-01T05:47:58.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170' 2024-04-01T05:47:58.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:47:58.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170 2-64424509461' 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397509 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397509 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397509' 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 210453397509 2024-04-01T05:47:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:47:59.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397509 -lt 210453397509 2024-04-01T05:47:59.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:59.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-04-01T05:47:59.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:59.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:47:59.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-04-01T05:47:59.410 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:59.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-04-01T05:47:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-04-01T05:47:59.412 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 249108103170 2024-04-01T05:47:59.412 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:47:59.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-04-01T05:47:59.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:47:59.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-04-01T05:47:59.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:47:59.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:47:59.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-04-01T05:47:59.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:47:59.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-04-01T05:47:59.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-04-01T05:47:59.697 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509461 2024-04-01T05:47:59.697 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:47:59.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-04-01T05:47:59.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:47:59.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:47:59.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:00.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:48:00.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:48:00.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:48:00.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:48:00.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:48:00.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:48:00.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:48:00.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:48:00.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:48:00.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:48:00.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:00.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-04-01T05:48:00.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:48:00.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:48:00.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-04-01T05:48:00.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:48:00.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:48:00.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:48:01.460 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:48:01.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:48:01.853 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:48:01.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:48:01.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:48:01.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:48:02.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-04-01T05:48:02.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-04-01T05:48:02.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:48:02.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-04-01T05:48:02.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-04-01T05:48:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-04-01T05:48:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:48:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:48:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:48:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:48:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:48:02.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:48:02.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:48:02.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:48:02.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:48:02.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:48:02.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:48:02.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:48:02.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:48:04.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:04.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:48:04.673 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:48:04.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:48:04.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:48:04.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:48:04.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:48:04.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:48:04.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:48:04.700 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:04.700+0000 7fe053d14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:04.703 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:04.703+0000 7fe053d14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:04.704 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:04.704+0000 7fe053d14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:04.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:48:04.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:48:04.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:48:04.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:48:04.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:48:04.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:04.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:48:04.968 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:48:04.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:04.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:05.220 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:05.220+0000 7fe053d14640 -1 Falling back to public interface 2024-04-01T05:48:05.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:06.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:06.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:06.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:48:06.251 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:06.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:06.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:06.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:07.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:07.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:07.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:48:07.545 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:48:07.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:07.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:07.611 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:07.611+0000 7fe053d14640 -1 osd.0 62 log_to_monitors true 2024-04-01T05:48:07.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:08.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:08.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:08.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:48:08.845 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:48:08.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:08.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/4155716420,v1:127.0.0.1:6803/4155716420] [v2:127.0.0.1:6804/4155716420,v1:127.0.0.1:6805/4155716420] exists,up ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:48:09.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:48:09.361 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:48:09.361 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:48:09.361 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:48:09.361 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:48:09.361 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:48:09.446 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:48:09.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:48:09.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:48:09.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:48:09.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:48:09.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:48:09.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:48:09.783 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:09.783 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:48:09.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:48:09.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:09.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:48:09.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-04-01T05:48:09.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-04-01T05:48:09.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-04-01T05:48:09.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:09.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:48:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-04-01T05:48:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-04-01T05:48:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-249108103173' 2024-04-01T05:48:10.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:10.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:48:10.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-04-01T05:48:10.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-04-01T05:48:10.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-249108103173 2-64424509465' 2024-04-01T05:48:10.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:10.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-04-01T05:48:10.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:10.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:48:10.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-04-01T05:48:10.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:10.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-04-01T05:48:10.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-04-01T05:48:10.150 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 283467841538 2024-04-01T05:48:10.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:10.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-04-01T05:48:10.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:48:11.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:48:11.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:11.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-04-01T05:48:11.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:11.742 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-04-01T05:48:11.742 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:11.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:48:11.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-04-01T05:48:11.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-04-01T05:48:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-04-01T05:48:11.745 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 249108103173 2024-04-01T05:48:11.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:48:12.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103173 -lt 249108103173 2024-04-01T05:48:12.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:12.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-04-01T05:48:12.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:12.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:48:12.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-04-01T05:48:12.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-04-01T05:48:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-04-01T05:48:12.030 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509465 2024-04-01T05:48:12.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:48:12.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-04-01T05:48:12.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:48:12.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:12.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:12.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:48:12.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:48:12.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:48:12.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:48:12.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:12.955 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:48:13.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:48:13.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:48:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:48:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:48:13.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:48:13.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:48:13.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:48:13.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:48:13.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:48:13.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:48:13.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:48:13.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:48:13.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:48:15.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:48:15.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:48:15.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:48:15.795 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.2 2024-04-01T05:48:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:48:15.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:48:15.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:48:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:48:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:48:15.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:48:15.822 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:15.821+0000 7fa39701e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:15.824 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:15.824+0000 7fa39701e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:15.825 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:15.825+0000 7fa39701e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:16.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:48:16.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:48:16.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:16.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:16.340 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:16.340+0000 7fa39701e640 -1 Falling back to public interface 2024-04-01T05:48:16.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:17.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:17.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:17.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:48:17.368 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:17.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:17.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:17.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:48:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:48:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:18.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:18.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:18.720+0000 7fa39701e640 -1 osd.2 69 log_to_monitors true 2024-04-01T05:48:19.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:20.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:20.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:20.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:48:20.083 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:48:20.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:20.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:20.427 INFO:tasks.workunit.client.0.smithi067.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/2624754474,v1:127.0.0.1:6819/2624754474] [v2:127.0.0.1:6820/2624754474,v1:127.0.0.1:6821/2624754474] exists,up 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 2024-04-01T05:48:20.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:48:20.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:48:20.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:48:20.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:48:20.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:48:20.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:48:20.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:48:20.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:48:20.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:48:20.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:48:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:48:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:48:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:48:20.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:20.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:48:20.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-04-01T05:48:20.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-04-01T05:48:20.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-04-01T05:48:20.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:20.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:48:21.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103176 2024-04-01T05:48:21.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103176 2024-04-01T05:48:21.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-249108103176' 2024-04-01T05:48:21.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:21.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:48:21.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-04-01T05:48:21.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-04-01T05:48:21.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-249108103176 2-317827579906' 2024-04-01T05:48:21.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:21.112 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-04-01T05:48:21.112 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:21.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:48:21.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-04-01T05:48:21.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:21.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-04-01T05:48:21.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-04-01T05:48:21.114 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 283467841541 2024-04-01T05:48:21.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:21.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-04-01T05:48:21.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:21.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103176 2024-04-01T05:48:21.408 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:21.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:48:21.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103176 2024-04-01T05:48:21.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:21.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103176 2024-04-01T05:48:21.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103176' 2024-04-01T05:48:21.411 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 249108103176 2024-04-01T05:48:21.411 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:48:21.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103176 -lt 249108103176 2024-04-01T05:48:21.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:21.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-04-01T05:48:21.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:21.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:48:21.697 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-04-01T05:48:21.697 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:21.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-04-01T05:48:21.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-04-01T05:48:21.698 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 317827579906 2024-04-01T05:48:21.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:48:21.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-04-01T05:48:21.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:48:21.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:21.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:48:22.332 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:48:22.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:48:22.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:48:22.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:22.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:48:22.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:48:22.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:48:23.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:48:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:48:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:48:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:48:25.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:48:25.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:48:25.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.2 2024-04-01T05:48:25.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:48:25.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:48:25.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:48:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:48:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:48:25.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:48:25.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:25.383+0000 7fa80b90d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:25.386 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:25.385+0000 7fa80b90d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:25.387 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:25.386+0000 7fa80b90d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:25.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:25.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:25.901+0000 7fa80b90d640 -1 Falling back to public interface 2024-04-01T05:48:25.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:26.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:26.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:26.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:48:26.932 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:26.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:26.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:27.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:28.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:28.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:28.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:48:28.230 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:48:28.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:28.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:28.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:28.240+0000 7fa80b90d640 -1 osd.2 76 log_to_monitors true 2024-04-01T05:48:28.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:29.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:29.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:29.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:48:29.528 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:48:29.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:29.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:48:29.925 INFO:tasks.workunit.client.0.smithi067.stderr:osd.2 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/36335249,v1:127.0.0.1:6819/36335249] [v2:127.0.0.1:6820/36335249,v1:127.0.0.1:6821/36335249] exists,up 274345d2-b3b0-4d7b-bb71-4c8b79c3bb51 2024-04-01T05:48:29.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:48:29.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:48:29.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:48:29.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:48:29.926 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:48:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:48:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:48:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:48:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:48:30.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:48:30.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:48:30.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:48:30.344 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:30.344 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:48:30.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:48:30.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:30.344 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:48:30.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841544 2024-04-01T05:48:30.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841544 2024-04-01T05:48:30.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544' 2024-04-01T05:48:30.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:30.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:48:30.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103179 2024-04-01T05:48:30.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103179 2024-04-01T05:48:30.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-249108103179' 2024-04-01T05:48:30.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:30.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:48:30.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-04-01T05:48:30.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-04-01T05:48:30.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841544 1-249108103179 2-347892350978' 2024-04-01T05:48:30.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:30.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841544 2024-04-01T05:48:30.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:30.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:48:30.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841544 2024-04-01T05:48:30.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:30.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841544 2024-04-01T05:48:30.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841544' 2024-04-01T05:48:30.690 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 283467841544 2024-04-01T05:48:30.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841544 2024-04-01T05:48:30.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:48:31.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:48:31.974 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:32.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841544 -lt 283467841544 2024-04-01T05:48:32.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:32.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103179 2024-04-01T05:48:32.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:32.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:48:32.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103179 2024-04-01T05:48:32.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:32.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103179 2024-04-01T05:48:32.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103179' 2024-04-01T05:48:32.273 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 249108103179 2024-04-01T05:48:32.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:48:32.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103179 -lt 249108103179 2024-04-01T05:48:32.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:32.599 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-04-01T05:48:32.599 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:32.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:48:32.601 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-04-01T05:48:32.601 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:32.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-04-01T05:48:32.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-04-01T05:48:32.603 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 347892350978 2024-04-01T05:48:32.603 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:48:32.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-04-01T05:48:32.886 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:48:32.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:32.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:33.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:48:33.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:48:33.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:48:33.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:48:33.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:33.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:33.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:48:33.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:48:33.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:48:33.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-04-01T05:48:33.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:48:33.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-04-01T05:48:33.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-04-01T05:48:33.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:48:33.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:48:33.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-04-01T05:48:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:48:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:48:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:48:34.330 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:48:34.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:48:34.823 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:48:34.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:48:34.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:48:34.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:48:35.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-04-01T05:48:35.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:48:35.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:48:35.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:2+0 records in 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:2+0 records out 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.793e-05 s, 60.3 MB/s 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:48:35.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:48:35.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:48:35.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:48:35.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:48:37.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:48:37.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:48:37.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:48:37.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:48:37.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:48:37.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:48:37.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:48:37.575 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:48:37.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:48:37.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:48:37.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:48:37.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:48:37.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:48:37.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:48:37.602 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:37.602+0000 7f89e1b26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:37.604 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:37.604+0000 7f89e1b26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:37.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:37.606+0000 7f89e1b26640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:48:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:48:37.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:37.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:48:38.119 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:38.119+0000 7f89e1b26640 -1 Falling back to public interface 2024-04-01T05:48:38.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:48:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:39.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:48:39.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:40.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:40.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:40.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:48:40.478 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:48:40.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:40.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:48:40.579 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:40.580+0000 7f89e1b26640 -1 osd.1 86 log_to_monitors true 2024-04-01T05:48:40.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:48:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:48:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [58,86) [v2:127.0.0.1:6810/2862137878,v1:127.0.0.1:6811/2862137878] [v2:127.0.0.1:6812/2862137878,v1:127.0.0.1:6813/2862137878] exists,up 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:48:42.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:48:42.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:48:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:48:42.215 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:48:42.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:48:42.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:48:42.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:48:42.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:48:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:48:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:48:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:48:42.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:48:42.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:48:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:48:42.583 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:48:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:48:42.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:42.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:48:42.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841547 2024-04-01T05:48:42.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841547 2024-04-01T05:48:42.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547' 2024-04-01T05:48:42.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:42.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:48:42.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-04-01T05:48:42.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-04-01T05:48:42.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547 1-390842023938' 2024-04-01T05:48:42.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:42.919 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:48:43.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350981 2024-04-01T05:48:43.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350981 2024-04-01T05:48:43.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841547 1-390842023938 2-347892350981' 2024-04-01T05:48:43.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:43.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841547 2024-04-01T05:48:43.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:43.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:48:43.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841547 2024-04-01T05:48:43.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:43.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841547 2024-04-01T05:48:43.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841547' 2024-04-01T05:48:43.027 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 283467841547 2024-04-01T05:48:43.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:43.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841546 -lt 283467841547 2024-04-01T05:48:43.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:48:44.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:48:44.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:44.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841548 -lt 283467841547 2024-04-01T05:48:44.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:44.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023938 2024-04-01T05:48:44.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:44.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:48:44.608 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023938 2024-04-01T05:48:44.608 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-04-01T05:48:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-04-01T05:48:44.609 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 390842023938 2024-04-01T05:48:44.609 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:48:44.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023938 -lt 390842023938 2024-04-01T05:48:44.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:44.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350981 2024-04-01T05:48:44.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:44.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:48:44.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350981 2024-04-01T05:48:44.895 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:44.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350981 2024-04-01T05:48:44.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350981' 2024-04-01T05:48:44.896 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 347892350981 2024-04-01T05:48:44.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:48:45.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350981 -lt 347892350981 2024-04-01T05:48:45.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:48:45.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:45.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:48:45.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:48:45.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:48:45.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:48:45.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:48:45.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:48:45.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:48:45.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:48:45.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:48:45.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:48:45.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:45.813 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:48:46.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-04-01T05:48:46.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:48:46.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:48:46.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:48:46.567 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:48:46.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:48:47.086 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:48:47.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:48:47.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:48:47.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:48:47.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:48:47.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:48:47.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:48:48.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:48:48.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:48:48.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:48:48.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:48:48.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:48:48.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:48:48.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:48:48.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:48:50.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:48:50.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:48:50.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:48:50.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:48:50.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:50.248+0000 7fcc1a919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:50.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:50.251+0000 7fcc1a919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:50.252 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:50.253+0000 7fcc1a919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:50.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:50.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:50.775+0000 7fcc1a919640 -1 Falling back to public interface 2024-04-01T05:48:50.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:51.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:51.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:51.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:48:51.804 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:51.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:51.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:52.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:53.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:53.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:53.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:48:53.096 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:48:53.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:53.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:53.169 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:48:53.169+0000 7fcc1a919640 -1 osd.0 95 log_to_monitors true 2024-04-01T05:48:53.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:48:54.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:48:54.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:48:54.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:48:54.467 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:48:54.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:48:54.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [66,95) [v2:127.0.0.1:6802/721037340,v1:127.0.0.1:6803/721037340] [v2:127.0.0.1:6804/721037340,v1:127.0.0.1:6805/721037340] exists,up ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:48:54.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:48:54.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:48:54.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:48:54.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:48:54.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:48:54.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:48:54.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:48:54.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:48:55.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:48:55.284 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:48:55.284 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:48:55.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:48:55.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:55.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:48:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-04-01T05:48:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-04-01T05:48:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602' 2024-04-01T05:48:55.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:55.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:48:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023941 2024-04-01T05:48:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023941 2024-04-01T05:48:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023941' 2024-04-01T05:48:55.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:48:55.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:48:55.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350985 2024-04-01T05:48:55.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350985 2024-04-01T05:48:55.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023941 2-347892350985' 2024-04-01T05:48:55.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729602 2024-04-01T05:48:55.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:55.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:48:55.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729602 2024-04-01T05:48:55.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:55.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-04-01T05:48:55.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-04-01T05:48:55.671 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 429496729602 2024-04-01T05:48:55.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:55.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 429496729602 2024-04-01T05:48:55.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:48:56.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:48:56.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:57.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 429496729602 2024-04-01T05:48:57.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:48:58.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:48:58.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:48:58.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729602 -lt 429496729602 2024-04-01T05:48:58.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:58.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023941 2024-04-01T05:48:58.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:58.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:48:58.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023941 2024-04-01T05:48:58.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:58.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023941 2024-04-01T05:48:58.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023941' 2024-04-01T05:48:58.544 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 390842023941 2024-04-01T05:48:58.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:48:58.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023942 -lt 390842023941 2024-04-01T05:48:58.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:48:58.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350985 2024-04-01T05:48:58.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:48:58.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:48:58.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350985 2024-04-01T05:48:58.833 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:48:58.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350985 2024-04-01T05:48:58.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-04-01T05:48:58.834 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 347892350985 2024-04-01T05:48:58.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:48:59.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350985 -lt 347892350985 2024-04-01T05:48:59.117 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:48:59.117 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:59.117 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:48:59.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:48:59.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:48:59.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:48:59.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:48:59.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:48:59.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:48:59.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:48:59.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:48:59.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:48:59.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:48:59.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:48:59.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:49:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:49:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:49:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:49:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:49:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:49:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:49:00.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:49:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:49:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:49:00.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:49:02.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:02.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:49:02.470 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:49:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:49:02.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:49:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:49:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:49:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:49:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:49:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:02.497+0000 7f0c7220c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:02.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:02.500+0000 7f0c7220c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:02.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:02.501+0000 7f0c7220c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:02.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:49:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:49:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:49:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:49:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:49:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:49:02.768 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:49:02.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:02.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:49:03.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:03.016+0000 7f0c7220c640 -1 Falling back to public interface 2024-04-01T05:49:03.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:04.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:04.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:04.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:49:04.048 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:49:04.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:04.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:49:04.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:05.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:05.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:05.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:49:05.341 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:49:05.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:05.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:49:05.396 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:05.396+0000 7f0c7220c640 -1 osd.1 102 log_to_monitors true 2024-04-01T05:49:05.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:49:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:49:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:06.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:49:07.000 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/1814221426,v1:127.0.0.1:6811/1814221426] [v2:127.0.0.1:6812/1814221426,v1:127.0.0.1:6813/1814221426] exists,up 62979ae6-1ee9-4705-8b55-9b221f3e1caf 2024-04-01T05:49:07.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:49:07.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:49:07.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:49:07.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:49:07.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:49:07.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:49:07.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:49:07.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:49:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:49:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:49:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:49:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:49:07.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:07.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:49:07.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729605 2024-04-01T05:49:07.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729605 2024-04-01T05:49:07.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605' 2024-04-01T05:49:07.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:07.577 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:49:07.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-04-01T05:49:07.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-04-01T05:49:07.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-459561500674' 2024-04-01T05:49:07.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:07.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:49:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350988 2024-04-01T05:49:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350988 2024-04-01T05:49:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-459561500674 2-347892350988' 2024-04-01T05:49:07.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:07.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729605 2024-04-01T05:49:07.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:07.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:49:07.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729605 2024-04-01T05:49:07.789 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729605 2024-04-01T05:49:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-04-01T05:49:07.790 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 429496729605 2024-04-01T05:49:07.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:49:08.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729605 2024-04-01T05:49:08.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:49:09.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:49:09.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:49:09.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729606 -lt 429496729605 2024-04-01T05:49:09.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:09.372 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500674 2024-04-01T05:49:09.372 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:09.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:49:09.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500674 2024-04-01T05:49:09.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:09.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-04-01T05:49:09.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-04-01T05:49:09.375 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 459561500674 2024-04-01T05:49:09.375 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:49:09.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500674 -lt 459561500674 2024-04-01T05:49:09.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:09.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350988 2024-04-01T05:49:09.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:09.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:49:09.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350988 2024-04-01T05:49:09.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:09.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350988 2024-04-01T05:49:09.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-04-01T05:49:09.687 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 347892350988 2024-04-01T05:49:09.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:49:09.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350989 -lt 347892350988 2024-04-01T05:49:09.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:49:09.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:09.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:10.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:49:10.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:49:10.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:49:10.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:49:10.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:49:10.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:49:10.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:49:10.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:49:10.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:49:10.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:49:10.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:10.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:10.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:49:10.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:49:10.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:49:10.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:49:10.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:49:10.968 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:49:11.446 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:49:11.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:49:11.847 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:49:11.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:49:11.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:49:11.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:49:12.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-04-01T05:49:12.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:49:12.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:49:12.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-04-01T05:49:12.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-04-01T05:49:12.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-04-01T05:49:12.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-04-01T05:49:12.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:49:12.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:49:12.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:49:12.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:49:12.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-04-01T05:49:14.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:49:14.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:49:14.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:49:14.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:49:14.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:49:14.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:49:14.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:49:14.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:49:14.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:49:14.499 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:49:14.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:49:14.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:49:14.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:49:14.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:49:14.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:49:14.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:49:14.526 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:14.524+0000 7f6101fdf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:14.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:14.527+0000 7f6101fdf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:14.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:14.528+0000 7f6101fdf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:14.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:15.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:15.047+0000 7f6101fdf640 -1 Falling back to public interface 2024-04-01T05:49:15.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:16.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:16.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:16.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:49:16.073 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:49:16.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:16.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:16.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:17.350 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:17.350+0000 7f6101fdf640 -1 osd.0 111 log_to_monitors true 2024-04-01T05:49:17.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:17.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:17.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:49:17.388 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:49:17.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:17.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 116 up_thru 107 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/1392388419,v1:127.0.0.1:6803/1392388419] [v2:127.0.0.1:6804/1392388419,v1:127.0.0.1:6805/1392388419] exists,up ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:49:17.798 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:49:17.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:49:17.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:49:17.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:49:17.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:49:17.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:49:17.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:49:17.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:49:17.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:49:17.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:49:18.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:49:18.246 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:49:18.246 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:49:18.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:49:18.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:18.246 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:49:18.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-04-01T05:49:18.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-04-01T05:49:18.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338' 2024-04-01T05:49:18.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:18.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:49:18.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500677 2024-04-01T05:49:18.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500677 2024-04-01T05:49:18.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-459561500677' 2024-04-01T05:49:18.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:18.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:49:18.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350992 2024-04-01T05:49:18.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350992 2024-04-01T05:49:18.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-459561500677 2-347892350992' 2024-04-01T05:49:18.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:18.805 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-498216206338 2024-04-01T05:49:18.805 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:18.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:49:18.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-498216206338 2024-04-01T05:49:18.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:18.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-04-01T05:49:18.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-04-01T05:49:18.807 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 498216206338 2024-04-01T05:49:18.807 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:49:19.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729606 -lt 498216206338 2024-04-01T05:49:19.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:49:20.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:49:20.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:49:20.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-04-01T05:49:20.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:20.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500677 2024-04-01T05:49:20.385 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:20.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:49:20.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500677 2024-04-01T05:49:20.387 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:20.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500677 2024-04-01T05:49:20.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500677' 2024-04-01T05:49:20.388 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 459561500677 2024-04-01T05:49:20.388 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:49:20.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500677 -lt 459561500677 2024-04-01T05:49:20.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:20.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350992 2024-04-01T05:49:20.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:20.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:49:20.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350992 2024-04-01T05:49:20.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:20.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350992 2024-04-01T05:49:20.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350992' 2024-04-01T05:49:20.675 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 347892350992 2024-04-01T05:49:20.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:49:20.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350992 -lt 347892350992 2024-04-01T05:49:20.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:49:20.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:20.961 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:21.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:49:21.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:49:21.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:49:21.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:49:21.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:49:21.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:49:21.319 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:49:21.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:49:21.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:49:21.599 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:49:21.599 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:21.599 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:21.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-04-01T05:49:21.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-04-01T05:49:21.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:49:21.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-04-01T05:49:21.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:49:21.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:49:22.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:49:22.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:49:22.054 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:49:22.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:49:22.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:49:22.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:49:22.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:49:22.055 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:49:22.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T05:49:22.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:49:22.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:22.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:49:22.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:49:23.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:49:23.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:49:23.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:49:23.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:49:23.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:49:23.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:49:23.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:49:23.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:49:23.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:49:23.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:49:23.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:49:23.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:49:23.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:49:23.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:49:23.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:49:23.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:23.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:49:23.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:49:23.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:49:24.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:49:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:49:26.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:49:26.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:49:26.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:49:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:49:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:49:26.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:49:26.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:49:26.877 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:26.877+0000 7fa3abcaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:26.879 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:26.879+0000 7fa3abcaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:26.880 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:26.880+0000 7fa3abcaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:27.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:27.394 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:27.394+0000 7fa3abcaa640 -1 Falling back to public interface 2024-04-01T05:49:27.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:28.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:28.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:28.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:49:28.454 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:49:28.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:28.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:28.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:29.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:49:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:49:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:29.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:29.771 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:29.771+0000 7fa3abcaa640 -1 osd.0 118 log_to_monitors true 2024-04-01T05:49:30.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:31.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:31.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:31.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:49:31.160 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:49:31.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:31.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/3455230977,v1:127.0.0.1:6803/3455230977] [v2:127.0.0.1:6804/3455230977,v1:127.0.0.1:6805/3455230977] exists,up ce2aed0c-9bdc-465a-ae6e-3d05b635ff5a 2024-04-01T05:49:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:49:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:49:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:49:31.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:49:31.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:49:31.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:49:31.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:49:31.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:49:31.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:49:31.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:49:31.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:49:31.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:49:31.845 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:49:31.845 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:49:31.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:49:31.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:31.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:49:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-04-01T05:49:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-04-01T05:49:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410' 2024-04-01T05:49:31.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:31.991 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:49:32.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500681 2024-04-01T05:49:32.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500681 2024-04-01T05:49:32.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-459561500681' 2024-04-01T05:49:32.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:49:32.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:49:32.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350995 2024-04-01T05:49:32.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350995 2024-04-01T05:49:32.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-459561500681 2-347892350995' 2024-04-01T05:49:32.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:32.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-528280977410 2024-04-01T05:49:32.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:32.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:49:32.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-528280977410 2024-04-01T05:49:32.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:32.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-04-01T05:49:32.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-04-01T05:49:32.256 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 528280977410 2024-04-01T05:49:32.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:49:32.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 528280977410 2024-04-01T05:49:32.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:49:33.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:49:33.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:49:33.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-04-01T05:49:33.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:33.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500681 2024-04-01T05:49:33.836 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:33.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:49:33.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500681 2024-04-01T05:49:33.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:33.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500681 2024-04-01T05:49:33.839 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500681' 2024-04-01T05:49:33.839 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 459561500681 2024-04-01T05:49:33.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:49:34.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500681 -lt 459561500681 2024-04-01T05:49:34.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:49:34.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350995 2024-04-01T05:49:34.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:49:34.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:49:34.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350995 2024-04-01T05:49:34.124 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:49:34.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350995 2024-04-01T05:49:34.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350995' 2024-04-01T05:49:34.125 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 347892350995 2024-04-01T05:49:34.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:49:34.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350995 -lt 347892350995 2024-04-01T05:49:34.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:49:34.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:34.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:49:34.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:49:35.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:49:35.051 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:49:35.052 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:49:35.052 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:49:35.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:49:35.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:49:35.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:49:35.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-04-01T05:49:35.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-04-01T05:49:35.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:49:35.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-04-01T05:49:35.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-04-01T05:49:35.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:49:35.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:35.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:49:35.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:35.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-04-01T05:49:36.249 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:36.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:49:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:36.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:37.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:37.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:37.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:49:37.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:37.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:49:37.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:37.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:49:37.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:37.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:38.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:38.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:38.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:49:38.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:38.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:49:38.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:38.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:49:39.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:39.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:40.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:40.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:40.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:49:40.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:40.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:49:40.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:40.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:49:40.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:40.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:41.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:41.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:49:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:49:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:49:41.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:49:36.325968+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:41.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:49:41.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-04-01T05:49:41.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-04-01T05:49:41.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-04-01T05:49:41.779 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T05:49:41.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-04-01T05:49:41.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-04-01T05:49:41.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-04-01T05:49:41.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-04-01T05:49:41.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:49:41.811 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:49:41.811 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:49:41.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:49:41.835 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:49:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:49:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:49:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T05:49:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:49:41.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:49:41.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:49:41.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:49:41.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-04-01T05:49:41.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-04-01T05:49:41.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-04-01T05:49:41.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-04-01T05:49:41.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:49:41.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:41.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:49:41.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:41.860 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:49:42.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:42.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-04-01T05:49:42.444 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-04-01T05:47:15.384124+0000 last_deep_scrub_stamp 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:49:42.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:42.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:49:42.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:42.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:49:42.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:42.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:43.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:43.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:43.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:49:43.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:43.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:49:43.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:43.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:49:44.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:44.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:45.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:45.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:45.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:49:45.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:45.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:49:45.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:45.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:49:45.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:45.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:46.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:46.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:49:46.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:46.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:49:46.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:46.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:49:46.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:47:15.384124+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:46.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:49:47.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:49:47.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:49:47.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:49:47.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:49:47.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:49:47.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:49:47.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:49:47.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:49:43.438104+0000 '>' 2024-04-01T05:47:15.384124+0000 2024-04-01T05:49:47.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:49:47.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-04-01T05:49:47.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-04-01T05:49:47.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-04-01T05:49:47.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T05:49:47.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-04-01T05:49:47.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-04-01T05:49:47.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:49:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:49:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:49:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T05:49:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:49:47.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:49:47.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:49:47.989 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:49:47.989 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:49:47.989 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:49:47.989 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:49:47.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:49:48.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:49:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:49:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-04-01T05:49:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-04-01T05:49:48.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-04-01T05:49:48.461 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' removed 2024-04-01T05:49:48.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:49:48.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:49:48.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:49:48.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:49:48.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:49:48.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:49:48.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:49:48.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:49:48.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:49:48.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:49:48.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:49:48.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:49:48.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:49:48.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:49:48.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:49:48.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:49:48.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:49:48.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:49:48.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:49:48.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:49:48.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:49:48.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:49:48.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:49:48.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:49:48.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:48.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:48.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:49:48.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:49:48.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:49:48.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:49:48.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:49:48.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:49:48.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:49:48.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:49:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:49:48.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:49:48.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:49:48.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:49:48.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:49:48.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:49:48.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:49:48.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:49:48.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:49:48.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:48.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:48.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:49:48.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:49:48.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:49:48.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:49:48.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:49:48.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:48.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:48.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-04-01T05:49:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-04-01T05:49:48.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:49:48.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:49:48.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:49:48.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:49:48.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:49:48.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:49.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T05:49:49.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:49:49.188 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:49:49.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:49:49.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:49:49.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:49:49.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:49:49.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:49:49.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:49:49.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:49:49.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-04-01T05:49:49.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:49:49.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:49:49.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:49:49.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:49:49.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:49:49.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:49:50.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:49:50.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:49:50.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:49:50.163 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:49:50.164 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:50.164 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:50.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:50.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:49:50.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:49:50.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-04-01T05:49:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-04-01T05:49:50.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:49:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:49:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:49:50.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 222075b5-81e3-4066-81b4-c13bf2b80e2a' 2024-04-01T05:49:50.199 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:49:50.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:49:50.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+SgpmjgbEDBAA6ZzMe/V8HUx6TmMSLgQncQ== 2024-04-01T05:49:50.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+SgpmjgbEDBAA6ZzMe/V8HUx6TmMSLgQncQ=="}' 2024-04-01T05:49:50.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 222075b5-81e3-4066-81b4-c13bf2b80e2a -i td/osd-scrub-repair/0/new.json 2024-04-01T05:49:50.419 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:49:50.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:49:50.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+SgpmjgbEDBAA6ZzMe/V8HUx6TmMSLgQncQ== --osd-uuid 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:49:50.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.457+0000 7f501fe0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:50.470 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.469+0000 7f501fe0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:50.471 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.470+0000 7f501fe0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:50.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.471+0000 7f501fe0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:49:50.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.471+0000 7f501fe0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:49:50.472 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.471+0000 7f501fe0e640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:49:50.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:50.561+0000 7f501fe0e640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:49:56.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:49:56.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:49:56.182 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:49:56.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:49:56.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:49:56.557 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:49:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:49:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:49:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:49:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:49:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:49:56.583 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:56.582+0000 7fa1e4e08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:56.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:56.584+0000 7fa1e4e08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:56.586 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:56.585+0000 7fa1e4e08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:49:56.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:49:56.842 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:49:56.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:56.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:57.104 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:57.103+0000 7fa1e4e08640 -1 Falling back to public interface 2024-04-01T05:49:57.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:58.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:58.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:58.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:49:58.124 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:49:58.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:58.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:58.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:49:59.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:49:59.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:49:59.416 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:49:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:49:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:49:59.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:49:59.437 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:49:59.437+0000 7fa1e4e08640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:49:59.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:00.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:00.856 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:50:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:50:00.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:00.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:01.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:02.212 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:50:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:50:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:02.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:03.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:03.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:03.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:50:03.507 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:50:03.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:03.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:03.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:04.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:04.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:04.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:50:04.802 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:50:04.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:04.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:05.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:06.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:06.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:06.187 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:50:06.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:50:06.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:06.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:06.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:07.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:50:07.483 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:50:07.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:07.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:07.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:08.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:08.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:08.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:50:08.777 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:50:08.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:08.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:09.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:10.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:10.179 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:50:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:50:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:10.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:11.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:11.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:11.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:50:11.475 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:50:11.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:11.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:12.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:13.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:13.044 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:50:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:50:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:13.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:13.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:14.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:50:14.581 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:50:14.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:14.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:15.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:16.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:16.297 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:50:16.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:16.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:50:16.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:16.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:16.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:17.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:17.757 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:50:17.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:17.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:50:17.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:17.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:18.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:19.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:19.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:19.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:50:19.345 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:50:19.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:19.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:20.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:21.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:21.116 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:50:21.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:21.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:50:21.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:21.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:50:21.426 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4246123926,v1:127.0.0.1:6803/4246123926] [v2:127.0.0.1:6804/4246123926,v1:127.0.0.1:6805/4246123926] exists,up 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:50:21.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:50:21.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:50:21.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d' 2024-04-01T05:50:21.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:50:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdSwpmn1muGhAALp8D7L6CbvrPODU5eteKQg== 2024-04-01T05:50:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdSwpmn1muGhAALp8D7L6CbvrPODU5eteKQg=="}' 2024-04-01T05:50:21.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d -i td/osd-scrub-repair/1/new.json 2024-04-01T05:50:21.934 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:50:21.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:50:21.946 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdSwpmn1muGhAALp8D7L6CbvrPODU5eteKQg== --osd-uuid c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:50:21.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:21.975+0000 7fe38a00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:22.018 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:22.018+0000 7fe38a00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:22.019 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:22.019+0000 7fe38a00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:22.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:22.020+0000 7fe38a00d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:50:22.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:22.020+0000 7fe38a00d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:50:22.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:22.020+0000 7fe38a00d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:50:22.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:22.088+0000 7fe38a00d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:50:28.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:50:28.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:50:28.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:50:28.262 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:50:28.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:50:28.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:50:28.627 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:50:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:50:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:50:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:50:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:50:28.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:28.653+0000 7f25802d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:28.655 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:28.655+0000 7f25802d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:28.656 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:28.656+0000 7f25802d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:50:28.928 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:50:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:29.170 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:29.170+0000 7f25802d9640 -1 Falling back to public interface 2024-04-01T05:50:29.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:30.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:30.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:30.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:50:30.213 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:50:30.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:30.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:30.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:31.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:50:31.510 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:50:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:31.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:31.575 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:31.575+0000 7f25802d9640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:50:31.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:32.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:32.807 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:50:32.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:32.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:50:32.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:32.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:33.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:34.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:50:34.104 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:50:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:34.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:34.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:35.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:35.400 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:50:35.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:50:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:35.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:35.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:36.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:36.695 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:50:36.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:36.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:50:36.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:36.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:36.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:37.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:37.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:37.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:50:37.993 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:50:37.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:37.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:38.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:39.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:39.380 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:50:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:50:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:39.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:39.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:40.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:40.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:40.686 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:50:40.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:50:40.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:40.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:40.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:41.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:41.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:41.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:50:41.983 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:50:41.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:41.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:42.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:43.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:43.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:43.389 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:50:43.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:50:43.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:43.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:43.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:44.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:44.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:44.902 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:50:44.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:50:44.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:44.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:45.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:46.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:46.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:46.585 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:50:46.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:50:46.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:46.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:47.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:48.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:48.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:48.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:50:48.607 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:50:48.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:48.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:50.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:50:50.418 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:50:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:51.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:52.187 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:50:52.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:50:52.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:52.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:53.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:50:54.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:50:54.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:50:54.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:50:54.018 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:50:54.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:50:54.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:50:54.315 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/623619452,v1:127.0.0.1:6811/623619452] [v2:127.0.0.1:6812/623619452,v1:127.0.0.1:6813/623619452] exists,up c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:50:54.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:50:54.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:50:54.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=451ccdb5-f1ad-481f-8c32-77b96e00ab02 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 451ccdb5-f1ad-481f-8c32-77b96e00ab02' 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 451ccdb5-f1ad-481f-8c32-77b96e00ab02 2024-04-01T05:50:54.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:50:54.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+SwpmGuQKFBAA3ER84fCrLEScD5eg+rj5jw== 2024-04-01T05:50:54.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+SwpmGuQKFBAA3ER84fCrLEScD5eg+rj5jw=="}' 2024-04-01T05:50:54.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 451ccdb5-f1ad-481f-8c32-77b96e00ab02 -i td/osd-scrub-repair/2/new.json 2024-04-01T05:50:54.774 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:50:54.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T05:50:54.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+SwpmGuQKFBAA3ER84fCrLEScD5eg+rj5jw== --osd-uuid 451ccdb5-f1ad-481f-8c32-77b96e00ab02 2024-04-01T05:50:54.814 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.814+0000 7f06be08b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:54.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.859+0000 7f06be08b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:54.859 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.860+0000 7f06be08b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:50:54.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.860+0000 7f06be08b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:50:54.860 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.860+0000 7f06be08b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:50:54.861 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.860+0000 7f06be08b640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T05:50:54.925 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:50:54.925+0000 7f06be08b640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T05:51:00.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T05:51:00.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:51:00.841 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T05:51:00.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T05:51:00.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:51:01.211 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T05:51:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T05:51:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:51:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:51:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:51:01.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:51:01.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:01.237+0000 7fbdddaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:51:01.239 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:01.239+0000 7fbdddaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:51:01.240 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:01.240+0000 7fbdddaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:51:01.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:51:01.494 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:51:01.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:01.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:01.757 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:01.757+0000 7fbdddaf9640 -1 Falling back to public interface 2024-04-01T05:51:01.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:02.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:02.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:02.780 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:51:02.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:51:02.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:02.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:03.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:04.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:04.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:04.072 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:51:04.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:51:04.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:04.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:04.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:04.109+0000 7fbdddaf9640 -1 osd.2 0 log_to_monitors true 2024-04-01T05:51:04.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:05.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:05.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:05.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:51:05.427 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:51:05.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:05.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:05.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:06.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:06.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:06.797 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:51:06.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:51:06.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:06.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:07.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:08.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:08.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:08.138 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:51:08.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:51:08.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:08.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:08.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:09.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:09.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:09.467 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:51:09.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:51:09.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:09.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:09.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:10.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:10.762 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:51:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:51:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:10.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:11.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:12.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:12.083 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:51:12.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:12.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:51:12.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:12.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:12.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:13.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:13.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:13.380 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:51:13.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:51:13.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:13.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:13.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:14.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:14.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:14.734 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:51:14.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:51:14.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:14.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:15.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:16.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:16.095 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:51:16.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:16.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:51:16.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:16.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:16.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:17.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:17.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:17.531 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:51:17.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:51:17.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:17.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:18.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:19.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:19.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:19.135 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:51:19.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:51:19.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:19.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:19.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:20.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:20.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:20.588 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:51:20.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:51:20.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:20.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:20.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:21.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:21.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:21.988 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:51:21.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:51:21.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:21.989 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:23.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:24.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:24.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:24.018 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:51:24.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:51:24.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:24.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:25.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:25.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:25.500 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:51:25.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:51:25.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:25.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:26.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:27.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:27.155 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:51:27.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:27.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:51:27.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:27.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:51:27.486 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2046388204,v1:127.0.0.1:6819/2046388204] [v2:127.0.0.1:6820/2046388204,v1:127.0.0.1:6821/2046388204] exists,up 451ccdb5-f1ad-481f-8c32-77b96e00ab02 2024-04-01T05:51:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:51:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:51:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:51:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-04-01T05:51:27.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:51:27.759 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:51:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:51:27.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:51:28.143 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:51:28.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:51:29.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:51:31.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-04-01T05:51:31.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-04-01T05:51:31.948 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' created 2024-04-01T05:51:31.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:51:32.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-04-01T05:51:32.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T05:51:32.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T05:51:32.961 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-04-01T05:51:32.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T05:51:32.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-04-01T05:51:33.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T05:51:33.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T05:51:34.202 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T05:51:34.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:51:35.216 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-04-01T05:51:35.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-04-01T05:51:35.608 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 3 allow_ec_overwrites to true 2024-04-01T05:51:35.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T05:51:35.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:51:35.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:51:35.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:51:35.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:51:35.621 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:51:35.621 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:51:35.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:51:35.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:51:35.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:51:35.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:51:35.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:51:35.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:51:35.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:51:35.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:51:35.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:51:36.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:51:36.038 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:51:36.038 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:51:36.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:51:36.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:36.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:51:36.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-04-01T05:51:36.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-04-01T05:51:36.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-04-01T05:51:36.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:36.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:51:36.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-04-01T05:51:36.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-04-01T05:51:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672977' 2024-04-01T05:51:36.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:36.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:51:36.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-04-01T05:51:36.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-04-01T05:51:36.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672977 2-64424509448' 2024-04-01T05:51:36.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:36.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-04-01T05:51:36.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:36.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:51:36.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-04-01T05:51:36.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:36.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-04-01T05:51:36.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-04-01T05:51:36.419 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836496 2024-04-01T05:51:36.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:51:36.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-04-01T05:51:36.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:36.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-04-01T05:51:36.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:36.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:51:36.779 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-04-01T05:51:36.779 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:36.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-04-01T05:51:36.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-01T05:51:36.781 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672977 2024-04-01T05:51:36.781 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:51:37.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-01T05:51:37.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:37.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-04-01T05:51:37.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:37.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:51:37.085 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-04-01T05:51:37.085 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:37.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-04-01T05:51:37.086 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509448 2024-04-01T05:51:37.087 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-01T05:51:37.087 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:51:37.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509448 2024-04-01T05:51:37.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:51:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:51:38.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:51:38.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509448 2024-04-01T05:51:38.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:51:38.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:51:38.686 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:51:39.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:51:39.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:51:39.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:51:39.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:51:39.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:51:39.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:51:39.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:51:39.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:51:39.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:51:39.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:51:39.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:51:39.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:51:39.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:51:39.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:51:39.758 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:51:40.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:51:40.040 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:51:40.040 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:51:40.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:51:40.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:40.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:51:40.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-04-01T05:51:40.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-04-01T05:51:40.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-04-01T05:51:40.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:40.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:51:40.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-04-01T05:51:40.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-04-01T05:51:40.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672979' 2024-04-01T05:51:40.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:40.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:51:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-04-01T05:51:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-04-01T05:51:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672979 2-64424509450' 2024-04-01T05:51:40.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:40.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-04-01T05:51:40.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:40.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:51:40.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-04-01T05:51:40.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:40.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-04-01T05:51:40.360 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836498 2024-04-01T05:51:40.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-01T05:51:40.361 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:51:40.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-01T05:51:40.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:40.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-04-01T05:51:40.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:40.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:51:40.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-04-01T05:51:40.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:40.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-04-01T05:51:40.648 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672979 2024-04-01T05:51:40.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-01T05:51:40.648 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:51:40.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-04-01T05:51:40.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:40.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-04-01T05:51:40.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:40.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:51:40.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-04-01T05:51:40.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:40.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-04-01T05:51:40.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-01T05:51:40.935 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509450 2024-04-01T05:51:40.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:51:41.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509450 2024-04-01T05:51:41.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:51:42.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:51:42.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:51:42.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-01T05:51:42.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:51:42.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:51:42.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:51:42.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:51:43.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:51:43.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:51:43.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:51:43.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:51:43.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:51:43.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:51:43.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:51:43.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-04-01T05:51:43.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:51:43.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:51:43.945 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:51:43.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:51:44.473 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:51:44.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:51:44.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:51:44.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:51:44.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-04-01T05:51:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:51:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:51:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-04-01T05:51:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-04-01T05:51:44.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:44.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:51:44.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:51:44.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:51:44.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:51:44.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:44.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:51:44.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:51:45.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:51:47.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:51:47.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:51:47.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:51:47.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:51:47.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:51:47.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:51:47.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:51:47.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:51:47.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:51:47.498 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:47.498+0000 7fe985613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:51:47.500 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:47.500+0000 7fe985613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:51:47.501 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:47.501+0000 7fe985613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:47.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:51:48.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:48.016+0000 7fe985613640 -1 Falling back to public interface 2024-04-01T05:51:48.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:49.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:49.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:49.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:51:49.049 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:51:49.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:49.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:51:49.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:50.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:51:50.361+0000 7fe985613640 -1 osd.1 36 log_to_monitors true 2024-04-01T05:51:50.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:50.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:50.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:51:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:51:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:51:50.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:51:51.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:51:51.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:51:51.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:51:51.669 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:51:51.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:51:51.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/661621802,v1:127.0.0.1:6811/661621802] [v2:127.0.0.1:6812/661621802,v1:127.0.0.1:6813/661621802] exists,up c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:51:52.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:51:52.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:51:52.029 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:51:52.029 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:51:52.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:51:52.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:51:52.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:51:52.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:51:52.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:51:52.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:51:52.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:51:52.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:51:52.116 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:51:52.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:51:52.478 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:51:52.478 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:51:52.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:51:52.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:52.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:51:52.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-04-01T05:51:52.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-04-01T05:51:52.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-04-01T05:51:52.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:52.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:51:52.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-04-01T05:51:52.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-04-01T05:51:52.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-176093659138' 2024-04-01T05:51:52.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:51:52.709 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:51:52.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-04-01T05:51:52.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-04-01T05:51:52.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-176093659138 2-64424509454' 2024-04-01T05:51:52.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:52.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-04-01T05:51:52.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:52.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:51:52.816 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-04-01T05:51:52.816 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:52.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-04-01T05:51:52.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-01T05:51:52.817 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836502 2024-04-01T05:51:52.818 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:51:53.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-04-01T05:51:53.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:51:54.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:51:54.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:51:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-04-01T05:51:54.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:51:55.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:51:55.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:51:55.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-01T05:51:55.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:55.802 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659138 2024-04-01T05:51:55.802 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:55.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:51:55.804 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659138 2024-04-01T05:51:55.804 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:55.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-04-01T05:51:55.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-04-01T05:51:55.805 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 176093659138 2024-04-01T05:51:55.805 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:51:56.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-04-01T05:51:56.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:51:56.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-04-01T05:51:56.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:51:56.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:51:56.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-04-01T05:51:56.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:51:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-04-01T05:51:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-01T05:51:56.094 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509454 2024-04-01T05:51:56.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:51:56.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-01T05:51:56.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:51:56.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:51:56.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:51:56.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:51:56.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:51:56.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:51:56.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:51:56.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:51:56.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:51:56.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:51:56.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:51:57.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:51:57.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:51:57.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:51:57.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:51:57.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:51:57.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:51:57.751 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:51:57.761 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:51:58.253 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:51:58.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:51:58.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:51:58.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:51:58.515 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-04-01T05:51:58.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-04-01T05:51:58.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:51:58.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:1+0 records in 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:1+0 records out 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.2758e-05 s, 32.6 MB/s 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:51:58.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:51:58.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:51:58.520 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:51:58.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:51:58.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:51:58.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:51:58.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:51:58.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:51:58.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:51:58.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:51:58.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:51:58.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:51:58.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:51:58.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:52:01.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:52:01.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:52:01.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:52:01.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:52:01.241 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:52:01.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:52:01.243 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:52:01.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:52:01.244 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:52:01.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:52:01.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:52:01.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:52:01.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:52:01.270 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:01.270+0000 7f3ba00f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:01.285 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:01.285+0000 7f3ba00f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:01.286 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:01.287+0000 7f3ba00f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:52:01.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:01.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:01.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:01.800+0000 7f3ba00f9640 -1 Falling back to public interface 2024-04-01T05:52:01.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:02.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:52:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:03.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:04.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:52:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:52:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:04.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:04.162 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:04.161+0000 7f3ba00f9640 -1 osd.0 45 log_to_monitors true 2024-04-01T05:52:04.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:05.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:52:05.524 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:52:05.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:05.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:05.856 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [5,45) [v2:127.0.0.1:6802/3487820319,v1:127.0.0.1:6803/3487820319] [v2:127.0.0.1:6804/3487820319,v1:127.0.0.1:6805/3487820319] exists,up 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:52:05.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:52:05.858 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:52:05.858 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:52:05.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:52:05.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:52:05.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:52:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:52:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:52:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:52:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:52:05.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:52:05.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:52:06.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:52:06.249 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:06.249 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:52:06.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:52:06.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:06.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:52:06.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364802 2024-04-01T05:52:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364802 2024-04-01T05:52:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364802' 2024-04-01T05:52:06.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:06.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:52:06.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659142 2024-04-01T05:52:06.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659142 2024-04-01T05:52:06.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364802 1-176093659142' 2024-04-01T05:52:06.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:06.461 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:52:06.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-04-01T05:52:06.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-04-01T05:52:06.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364802 1-176093659142 2-64424509457' 2024-04-01T05:52:06.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:06.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-214748364802 2024-04-01T05:52:06.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:06.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:52:06.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-214748364802 2024-04-01T05:52:06.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:06.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364802 2024-04-01T05:52:06.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 214748364802' 2024-04-01T05:52:06.570 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 214748364802 2024-04-01T05:52:06.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:06.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 214748364802 2024-04-01T05:52:06.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:07.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:52:07.859 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:08.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 214748364802 2024-04-01T05:52:08.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:09.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:52:09.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:09.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364802 -lt 214748364802 2024-04-01T05:52:09.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:09.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659142 2024-04-01T05:52:09.449 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:09.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:52:09.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659142 2024-04-01T05:52:09.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:09.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659142 2024-04-01T05:52:09.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659142' 2024-04-01T05:52:09.453 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 176093659142 2024-04-01T05:52:09.453 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:52:09.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659142 -lt 176093659142 2024-04-01T05:52:09.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:09.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-04-01T05:52:09.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:09.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:52:09.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-04-01T05:52:09.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-04-01T05:52:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-01T05:52:09.739 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509457 2024-04-01T05:52:09.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:52:10.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509457 2024-04-01T05:52:10.021 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:52:10.021 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:10.021 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:10.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:52:10.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:52:10.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:52:10.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:52:10.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:52:10.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:52:10.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:52:10.378 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:52:10.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:52:10.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:52:10.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:10.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-04-01T05:52:11.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:52:11.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:52:11.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:52:11.520 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:52:11.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:52:11.914 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:52:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:52:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:52:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:52:12.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-04-01T05:52:12.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:52:12.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:52:12.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:52:12.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:52:12.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:52:12.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:52:12.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-04-01T05:52:12.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:52:12.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:52:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:52:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:52:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:52:12.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-04-01T05:52:14.145 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-04-01T05:52:14.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:52:14.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:52:14.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:52:14.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:52:14.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:52:14.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:52:14.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:52:14.669 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:52:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:52:14.670 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:52:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:52:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:52:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:52:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:52:14.696 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:14.696+0000 7f9e640bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:14.698 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:14.698+0000 7f9e640bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:14.699 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:14.699+0000 7f9e640bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:14.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:52:14.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:14.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:52:15.218 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:15.218+0000 7f9e640bf640 -1 Falling back to public interface 2024-04-01T05:52:15.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:16.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:16.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:16.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:52:16.245 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:16.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:16.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:52:16.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:17.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:17.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:17.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:52:17.578 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:52:17.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:17.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:52:17.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:17.605+0000 7f9e640bf640 -1 osd.1 54 log_to_monitors true 2024-04-01T05:52:17.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:18.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:18.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:18.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:52:18.982 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:52:18.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:18.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 59 up_thru 59 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/3708023994,v1:127.0.0.1:6811/3708023994] [v2:127.0.0.1:6812/3708023994,v1:127.0.0.1:6813/3708023994] exists,up c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:52:19.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:52:19.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:52:19.276 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:52:19.276 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:52:19.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:52:19.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:52:19.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:52:19.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:52:19.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:52:19.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:52:19.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:52:19.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:52:19.361 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:52:19.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:52:19.676 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:19.677 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:52:19.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:52:19.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:19.677 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:52:19.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-04-01T05:52:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-04-01T05:52:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364806' 2024-04-01T05:52:19.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:19.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:52:19.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-04-01T05:52:19.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-04-01T05:52:19.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364806 1-253403070466' 2024-04-01T05:52:19.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:19.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:52:19.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-04-01T05:52:19.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-04-01T05:52:19.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-214748364806 1-253403070466 2-64424509461' 2024-04-01T05:52:19.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:19.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-214748364806 2024-04-01T05:52:19.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:19.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:52:19.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-214748364806 2024-04-01T05:52:19.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:19.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-04-01T05:52:19.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 214748364806' 2024-04-01T05:52:19.996 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 214748364806 2024-04-01T05:52:19.996 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:20.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364804 -lt 214748364806 2024-04-01T05:52:20.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:21.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:52:21.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:21.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-04-01T05:52:21.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:21.589 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070466 2024-04-01T05:52:21.590 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:21.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:52:21.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070466 2024-04-01T05:52:21.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:21.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-04-01T05:52:21.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070466' 2024-04-01T05:52:21.594 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 253403070466 2024-04-01T05:52:21.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:52:21.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-04-01T05:52:21.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:21.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-04-01T05:52:21.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:21.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:52:21.925 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-04-01T05:52:21.925 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:21.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-04-01T05:52:21.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-04-01T05:52:21.926 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509461 2024-04-01T05:52:21.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:52:22.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-04-01T05:52:22.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:52:22.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:22.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:22.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:52:22.610 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:52:22.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:52:22.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:52:22.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:52:22.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:52:22.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:52:22.611 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:52:22.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:52:22.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:52:22.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:22.894 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:23.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:52:23.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:52:23.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:52:23.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:52:23.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-04-01T05:52:23.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-04-01T05:52:23.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:52:23.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:52:23.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-04-01T05:52:23.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:52:23.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:52:23.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:52:23.666 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:52:23.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:52:24.203 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:52:24.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:52:24.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:52:24.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:52:24.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-04-01T05:52:24.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-04-01T05:52:24.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:52:24.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-04-01T05:52:24.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-04-01T05:52:24.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-04-01T05:52:24.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:52:24.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:52:24.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:52:25.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:52:25.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:52:25.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:52:25.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:52:25.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:52:25.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:52:25.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:52:25.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:52:27.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:27.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:52:27.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:52:27.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:52:27.523 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:52:27.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:52:27.523 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:52:27.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:52:27.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:52:27.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:52:27.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:52:27.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:27.549+0000 7fdfec868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:27.552 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:27.552+0000 7fdfec868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:27.554+0000 7fdfec868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:27.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:52:27.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:52:27.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:52:27.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:52:27.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:52:27.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:27.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:52:27.819 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:52:27.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:27.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:28.068 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:28.068+0000 7fdfec868640 -1 Falling back to public interface 2024-04-01T05:52:28.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:29.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:29.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:29.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:52:29.107 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:29.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:29.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:29.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:30.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:30.379+0000 7fdfec868640 -1 osd.0 63 log_to_monitors true 2024-04-01T05:52:30.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:30.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:30.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:52:30.401 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:52:30.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:30.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:30.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:31.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:52:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:52:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:31.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:52:32.286 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 68 up_thru 68 down_at 64 last_clean_interval [50,63) [v2:127.0.0.1:6802/2639042960,v1:127.0.0.1:6803/2639042960] [v2:127.0.0.1:6804/2639042960,v1:127.0.0.1:6805/2639042960] exists,up 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:52:32.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:52:32.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:52:32.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:52:32.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:52:32.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:52:32.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:52:32.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:52:32.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:52:32.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:52:32.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:52:32.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:52:32.815 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:32.815 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:52:32.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:52:32.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:32.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:52:32.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-04-01T05:52:32.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-04-01T05:52:32.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776130' 2024-04-01T05:52:32.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:32.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:52:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-04-01T05:52:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-04-01T05:52:33.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776130 1-253403070470' 2024-04-01T05:52:33.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:33.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:52:33.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-04-01T05:52:33.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-04-01T05:52:33.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776130 1-253403070470 2-64424509465' 2024-04-01T05:52:33.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:33.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776130 2024-04-01T05:52:33.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:33.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:52:33.132 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776130 2024-04-01T05:52:33.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:33.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-04-01T05:52:33.134 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776130' 2024-04-01T05:52:33.134 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 292057776130 2024-04-01T05:52:33.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 292057776130 2024-04-01T05:52:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:34.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:52:34.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:34.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776130 -lt 292057776130 2024-04-01T05:52:34.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:34.714 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070470 2024-04-01T05:52:34.714 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:34.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:52:34.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070470 2024-04-01T05:52:34.715 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:34.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-04-01T05:52:34.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070470' 2024-04-01T05:52:34.716 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 253403070470 2024-04-01T05:52:34.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:52:34.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070470 -lt 253403070470 2024-04-01T05:52:35.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:35.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-04-01T05:52:35.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:35.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:52:35.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-04-01T05:52:35.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:35.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-04-01T05:52:35.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-04-01T05:52:35.003 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 64424509465 2024-04-01T05:52:35.003 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:52:35.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-04-01T05:52:35.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:52:35.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:35.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:35.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:52:35.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:52:35.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:52:35.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:52:35.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:52:35.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:52:35.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:52:35.672 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:52:35.947 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:52:35.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:52:35.947 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:35.948 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:36.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:52:36.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:52:36.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:52:36.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:52:36.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:52:36.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:52:36.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-04-01T05:52:38.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:52:38.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:52:38.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:52:38.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:52:38.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:52:38.775 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:52:38.776 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.2 2024-04-01T05:52:38.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:52:38.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:52:38.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:52:38.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:52:38.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:52:38.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:52:38.804 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:38.804+0000 7f8a4bcc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:38.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:38.806+0000 7f8a4bcc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:38.807 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:38.808+0000 7f8a4bcc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:39.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:39.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:39.329+0000 7f8a4bcc6640 -1 Falling back to public interface 2024-04-01T05:52:39.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:52:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:40.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:40.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:52:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:52:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:41.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:41.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:41.798+0000 7f8a4bcc6640 -1 osd.2 71 log_to_monitors true 2024-04-01T05:52:41.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:42.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:42.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:42.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:52:42.987 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:52:42.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:42.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:43.479 INFO:tasks.workunit.client.0.smithi067.stderr:osd.2 up in weight 1 up_from 76 up_thru 76 down_at 72 last_clean_interval [15,71) [v2:127.0.0.1:6818/2903932296,v1:127.0.0.1:6819/2903932296] [v2:127.0.0.1:6820/2903932296,v1:127.0.0.1:6821/2903932296] exists,up 451ccdb5-f1ad-481f-8c32-77b96e00ab02 2024-04-01T05:52:43.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:52:43.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:52:43.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:52:43.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:52:43.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:52:43.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:52:43.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:52:43.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:52:43.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:52:43.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:52:43.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:52:43.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:52:43.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:52:43.848 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:43.848 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:52:43.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:52:43.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:43.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:52:43.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776133 2024-04-01T05:52:43.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776133 2024-04-01T05:52:43.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776133' 2024-04-01T05:52:43.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:43.953 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:52:44.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070473 2024-04-01T05:52:44.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070473 2024-04-01T05:52:44.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776133 1-253403070473' 2024-04-01T05:52:44.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:44.058 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:52:44.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-04-01T05:52:44.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-04-01T05:52:44.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776133 1-253403070473 2-326417514498' 2024-04-01T05:52:44.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:44.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776133 2024-04-01T05:52:44.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:44.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:52:44.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776133 2024-04-01T05:52:44.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776133 2024-04-01T05:52:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776133' 2024-04-01T05:52:44.167 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 292057776133 2024-04-01T05:52:44.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:44.448 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776132 -lt 292057776133 2024-04-01T05:52:44.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:45.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:52:45.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:45.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776133 -lt 292057776133 2024-04-01T05:52:45.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:45.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070473 2024-04-01T05:52:45.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:45.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:52:45.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070473 2024-04-01T05:52:45.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:45.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070473 2024-04-01T05:52:45.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070473' 2024-04-01T05:52:45.746 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 253403070473 2024-04-01T05:52:45.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:52:46.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070473 -lt 253403070473 2024-04-01T05:52:46.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:46.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514498 2024-04-01T05:52:46.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:46.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:52:46.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514498 2024-04-01T05:52:46.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:46.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-04-01T05:52:46.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-04-01T05:52:46.031 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 326417514498 2024-04-01T05:52:46.031 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:52:46.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514498 -lt 326417514498 2024-04-01T05:52:46.317 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:52:46.317 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:46.317 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:52:46.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:52:46.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:52:46.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:52:46.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:46.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:52:47.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:52:47.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:52:47.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:52:49.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:52:49.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:52:49.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T05:52:49.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T05:52:49.692 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.2 2024-04-01T05:52:49.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:52:49.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T05:52:49.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T05:52:49.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:52:49.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:52:49.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:52:49.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:49.719+0000 7f531c413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:49.722 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:49.721+0000 7f531c413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:49.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:49.723+0000 7f531c413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:52:49.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:49.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:50.238 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:50.237+0000 7f531c413640 -1 Falling back to public interface 2024-04-01T05:52:50.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:51.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:51.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:51.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:52.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:52.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:52.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:52:52.570 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:52:52.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:52.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:52.708 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:52:52.707+0000 7f531c413640 -1 osd.2 78 log_to_monitors true 2024-04-01T05:52:52.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:52:53.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:52:53.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:52:53.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:52:53.908 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:52:53.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:52:53.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T05:52:54.233 INFO:tasks.workunit.client.0.smithi067.stderr:osd.2 up in weight 1 up_from 83 up_thru 83 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6818/1464582233,v1:127.0.0.1:6819/1464582233] [v2:127.0.0.1:6820/1464582233,v1:127.0.0.1:6821/1464582233] exists,up 451ccdb5-f1ad-481f-8c32-77b96e00ab02 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:52:54.234 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:52:54.235 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:52:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:52:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:52:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:52:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:52:54.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:52:54.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:52:54.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:52:54.602 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:52:54.602 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:52:54.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:52:54.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:54.602 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:52:54.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776136 2024-04-01T05:52:54.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776136 2024-04-01T05:52:54.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776136' 2024-04-01T05:52:54.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:54.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:52:54.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-04-01T05:52:54.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-04-01T05:52:54.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776136 1-253403070476' 2024-04-01T05:52:54.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:52:54.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:52:54.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-04-01T05:52:54.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-04-01T05:52:54.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776136 1-253403070476 2-356482285570' 2024-04-01T05:52:54.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:54.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776136 2024-04-01T05:52:54.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:54.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:52:54.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776136 2024-04-01T05:52:54.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:54.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776136 2024-04-01T05:52:54.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776136' 2024-04-01T05:52:54.930 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 292057776136 2024-04-01T05:52:54.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:55.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776135 -lt 292057776136 2024-04-01T05:52:55.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:56.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:52:56.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:56.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776135 -lt 292057776136 2024-04-01T05:52:56.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:52:57.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:52:57.537 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:52:57.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776137 -lt 292057776136 2024-04-01T05:52:57.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:57.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-253403070476 2024-04-01T05:52:57.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:57.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:52:57.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-253403070476 2024-04-01T05:52:57.834 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:57.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-04-01T05:52:57.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 253403070476' 2024-04-01T05:52:57.835 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 253403070476 2024-04-01T05:52:57.835 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:52:58.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070476 -lt 253403070476 2024-04-01T05:52:58.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:52:58.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285570 2024-04-01T05:52:58.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:52:58.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:52:58.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285570 2024-04-01T05:52:58.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:52:58.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-04-01T05:52:58.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285570' 2024-04-01T05:52:58.123 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 356482285570 2024-04-01T05:52:58.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:52:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-04-01T05:52:58.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:52:58.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:58.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:52:58.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:52:58.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:52:59.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:52:59.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:52:59.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:52:59.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:52:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:52:59.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:52:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:52:59.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:52:59.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:52:59.822 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:52:59.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:53:00.332 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:53:00.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:53:00.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:53:00.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:53:00.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-04-01T05:53:00.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:53:00.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:53:00.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-04-01T05:53:00.548 INFO:tasks.workunit.client.0.smithi067.stderr:2+0 records in 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:2+0 records out 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.7322e-05 s, 60.8 MB/s 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:53:00.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:53:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:53:00.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:53:00.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:53:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:53:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:53:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:53:00.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:53:00.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:53:03.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:53:03.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:53:03.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:53:03.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:53:03.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:53:03.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:53:03.183 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:53:03.184 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:53:03.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:53:03.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:53:03.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:53:03.185 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:53:03.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:53:03.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:53:03.212 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:03.212+0000 7fa93060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:03.214 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:03.214+0000 7fa93060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:03.215 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:03.215+0000 7fa93060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:53:03.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:03.514 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:03.737 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:03.737+0000 7fa93060f640 -1 Falling back to public interface 2024-04-01T05:53:03.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:04.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:04.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:04.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:53:04.795 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:04.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:04.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:05.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:06.072 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:06.072+0000 7fa93060f640 -1 osd.1 87 log_to_monitors true 2024-04-01T05:53:06.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:06.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:06.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:53:06.091 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:53:06.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:06.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:06.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:53:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:53:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [59,87) [v2:127.0.0.1:6810/527833567,v1:127.0.0.1:6811/527833567] [v2:127.0.0.1:6812/527833567,v1:127.0.0.1:6813/527833567] exists,up c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:53:07.739 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:53:07.740 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:53:07.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:53:07.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:53:07.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:53:07.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:53:07.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:53:07.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:53:07.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:53:07.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:53:07.826 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:53:08.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:53:08.129 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:08.129 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:53:08.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:53:08.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:08.129 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:53:08.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776140 2024-04-01T05:53:08.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776140 2024-04-01T05:53:08.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776140' 2024-04-01T05:53:08.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:08.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:53:08.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-04-01T05:53:08.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-04-01T05:53:08.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776140 1-395136991234' 2024-04-01T05:53:08.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:08.367 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:53:08.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285574 2024-04-01T05:53:08.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285574 2024-04-01T05:53:08.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-292057776140 1-395136991234 2-356482285574' 2024-04-01T05:53:08.473 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:08.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-292057776140 2024-04-01T05:53:08.473 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:08.474 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:53:08.474 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-292057776140 2024-04-01T05:53:08.475 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:08.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776140 2024-04-01T05:53:08.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 292057776140' 2024-04-01T05:53:08.476 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 292057776140 2024-04-01T05:53:08.476 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:08.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776140 -lt 292057776140 2024-04-01T05:53:08.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:08.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991234 2024-04-01T05:53:08.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:08.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:53:08.761 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991234 2024-04-01T05:53:08.761 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:08.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-04-01T05:53:08.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-04-01T05:53:08.762 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 395136991234 2024-04-01T05:53:08.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:09.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 395136991234 2024-04-01T05:53:09.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:10.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:53:10.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:10.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 395136991234 2024-04-01T05:53:10.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:11.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:53:11.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:11.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-04-01T05:53:11.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:11.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285574 2024-04-01T05:53:11.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:11.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:53:11.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285574 2024-04-01T05:53:11.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:11.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285574 2024-04-01T05:53:11.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285574' 2024-04-01T05:53:11.639 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 356482285574 2024-04-01T05:53:11.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:53:11.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285574 -lt 356482285574 2024-04-01T05:53:11.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:53:11.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:11.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:53:12.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:53:12.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:53:12.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:53:12.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:12.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-04-01T05:53:12.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:53:12.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:53:12.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:53:13.408 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:53:13.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:53:13.818 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:53:13.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:53:13.829 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:53:13.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:53:14.089 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-04-01T05:53:14.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-04-01T05:53:14.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:53:14.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:53:14.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:53:14.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:53:14.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-04-01T05:53:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:53:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:53:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:53:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:53:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:53:16.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:53:16.538 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:53:16.539 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:53:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:53:16.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:53:16.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:53:16.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:53:16.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:53:16.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:53:16.567 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:16.567+0000 7fab8e138640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:16.569 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:16.569+0000 7fab8e138640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:16.570 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:16.570+0000 7fab8e138640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:16.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:53:16.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:17.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:17.086+0000 7fab8e138640 -1 Falling back to public interface 2024-04-01T05:53:17.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:18.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:18.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:18.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:53:18.117 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:18.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:18.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:18.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:19.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:19.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:19.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:53:19.408 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:53:19.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:19.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:19.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:19.594+0000 7fab8e138640 -1 osd.0 96 log_to_monitors true 2024-04-01T05:53:19.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:53:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:53:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:20.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:21.048 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [68,96) [v2:127.0.0.1:6802/3898106798,v1:127.0.0.1:6803/3898106798] [v2:127.0.0.1:6804/3898106798,v1:127.0.0.1:6805/3898106798] exists,up 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:53:21.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:53:21.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:53:21.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:53:21.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:53:21.049 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:53:21.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:53:21.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:53:21.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:53:21.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:53:21.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:53:21.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:53:21.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:53:21.417 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:21.417 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:53:21.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:53:21.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:21.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:53:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-04-01T05:53:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-04-01T05:53:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898' 2024-04-01T05:53:21.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:21.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:53:21.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991238 2024-04-01T05:53:21.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991238 2024-04-01T05:53:21.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991238' 2024-04-01T05:53:21.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:21.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:53:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285577 2024-04-01T05:53:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285577 2024-04-01T05:53:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991238 2-356482285577' 2024-04-01T05:53:21.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:21.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696898 2024-04-01T05:53:21.787 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:21.788 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:53:21.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696898 2024-04-01T05:53:21.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-04-01T05:53:21.789 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-04-01T05:53:21.790 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 433791696898 2024-04-01T05:53:21.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 433791696898 2024-04-01T05:53:22.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:23.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:53:23.072 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-04-01T05:53:23.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:23.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991238 2024-04-01T05:53:23.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:23.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:53:23.382 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991238 2024-04-01T05:53:23.382 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:23.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991238 2024-04-01T05:53:23.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991238' 2024-04-01T05:53:23.383 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 395136991238 2024-04-01T05:53:23.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:23.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991238 -lt 395136991238 2024-04-01T05:53:23.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:23.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285577 2024-04-01T05:53:23.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:23.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:53:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285577 2024-04-01T05:53:23.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:23.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285577 2024-04-01T05:53:23.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285577' 2024-04-01T05:53:23.673 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 356482285577 2024-04-01T05:53:23.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:53:23.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285577 -lt 356482285577 2024-04-01T05:53:23.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:53:23.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:23.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:24.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:53:24.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:53:24.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:53:24.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:53:24.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:24.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:53:24.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:53:24.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:53:25.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:53:27.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:53:27.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:53:27.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:53:27.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:53:27.371 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:27.371+0000 7fa8932a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:27.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:27.373+0000 7fa8932a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:27.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:27.375+0000 7fa8932a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:27.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:53:27.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:53:27.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:53:27.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:53:27.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:53:27.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:27.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:53:27.639 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:53:27.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:27.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:27.892 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:27.892+0000 7fa8932a8640 -1 Falling back to public interface 2024-04-01T05:53:27.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:53:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:29.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:53:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:53:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:30.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:30.235 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:30.235+0000 7fa8932a8640 -1 osd.1 103 log_to_monitors true 2024-04-01T05:53:30.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:31.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:31.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:31.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:53:31.653 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:53:31.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:31.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:53:32.015 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [92,103) [v2:127.0.0.1:6810/1017828937,v1:127.0.0.1:6811/1017828937] [v2:127.0.0.1:6812/1017828937,v1:127.0.0.1:6813/1017828937] exists,up c8a9bc8d-5e3a-48a6-9c1b-ade64e2cc14d 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:53:32.016 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:53:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:53:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:53:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:53:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:53:32.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:53:32.103 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:53:32.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:53:32.539 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:32.539 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:53:32.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:53:32.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:32.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:53:32.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-04-01T05:53:32.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-04-01T05:53:32.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901' 2024-04-01T05:53:32.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:32.680 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:53:32.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-04-01T05:53:32.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-04-01T05:53:32.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970' 2024-04-01T05:53:32.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:32.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:53:32.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285580 2024-04-01T05:53:32.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285580 2024-04-01T05:53:32.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970 2-356482285580' 2024-04-01T05:53:32.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:32.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696901 2024-04-01T05:53:32.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:32.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:53:32.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696901 2024-04-01T05:53:32.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:32.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-04-01T05:53:32.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-04-01T05:53:32.892 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 433791696901 2024-04-01T05:53:32.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:33.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696901 2024-04-01T05:53:33.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:34.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:53:34.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696901 2024-04-01T05:53:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:35.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:53:35.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-04-01T05:53:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:35.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467970 2024-04-01T05:53:35.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:53:35.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467970 2024-04-01T05:53:35.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-04-01T05:53:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-04-01T05:53:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 463856467970 2024-04-01T05:53:35.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:36.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-04-01T05:53:36.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:36.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285580 2024-04-01T05:53:36.069 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:36.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:53:36.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285580 2024-04-01T05:53:36.071 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:36.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285580 2024-04-01T05:53:36.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285580' 2024-04-01T05:53:36.072 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 356482285580 2024-04-01T05:53:36.073 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:53:36.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285581 -lt 356482285580 2024-04-01T05:53:36.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:53:36.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:36.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:36.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:53:36.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:53:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:53:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:53:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:53:36.710 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:53:36.711 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:53:36.711 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:53:36.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:53:36.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:53:36.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:36.988 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-04-01T05:53:37.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:53:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T05:53:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-04-01T05:53:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:53:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:53:37.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:53:37.757 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:53:37.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:53:38.260 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:53:38.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:53:38.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:53:38.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:53:38.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-04-01T05:53:38.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-04-01T05:53:38.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-04-01T05:53:38.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-04-01T05:53:38.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-04-01T05:53:38.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-04-01T05:53:38.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-04-01T05:53:38.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:53:38.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:53:38.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:53:38.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-04-01T05:53:38.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:53:38.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:53:38.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:53:38.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:53:38.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:53:38.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-04-01T05:53:40.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:53:40.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:53:40.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:53:40.749 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:53:40.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:53:40.750 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:53:40.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:53:40.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:53:40.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:53:40.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:53:40.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:40.776+0000 7f1f48848640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:40.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:40.778+0000 7f1f48848640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:40.780 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:40.780+0000 7f1f48848640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:41.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:53:41.046 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:53:41.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:41.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:41.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:41.301+0000 7f1f48848640 -1 Falling back to public interface 2024-04-01T05:53:41.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:53:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:42.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:42.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:43.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:43.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:43.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:53:43.642 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:53:43.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:43.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:43.723 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:43.723+0000 7f1f48848640 -1 osd.0 113 log_to_monitors true 2024-04-01T05:53:43.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:44.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:44.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:44.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:53:44.996 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:53:44.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:44.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 118 up_thru 118 down_at 114 last_clean_interval [101,113) [v2:127.0.0.1:6802/416651608,v1:127.0.0.1:6803/416651608] [v2:127.0.0.1:6804/416651608,v1:127.0.0.1:6805/416651608] exists,up 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:53:45.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:53:45.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:53:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:53:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:53:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:53:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:53:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:53:45.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:53:45.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:53:45.749 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:45.749 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:53:45.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:53:45.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:45.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:53:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-04-01T05:53:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-04-01T05:53:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-506806140930' 2024-04-01T05:53:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:45.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:53:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467974 2024-04-01T05:53:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467974 2024-04-01T05:53:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-506806140930 1-463856467974' 2024-04-01T05:53:45.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:45.959 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:53:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285584 2024-04-01T05:53:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285584 2024-04-01T05:53:46.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-506806140930 1-463856467974 2-356482285584' 2024-04-01T05:53:46.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:46.065 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-506806140930 2024-04-01T05:53:46.065 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:46.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:53:46.066 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-506806140930 2024-04-01T05:53:46.066 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:46.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-04-01T05:53:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 506806140930' 2024-04-01T05:53:46.068 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 506806140930 2024-04-01T05:53:46.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:46.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 506806140930 2024-04-01T05:53:46.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:47.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:53:47.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:47.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140930 -lt 506806140930 2024-04-01T05:53:47.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:47.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467974 2024-04-01T05:53:47.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467974 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467974 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467974' 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 463856467974 2024-04-01T05:53:47.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:47.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467974 -lt 463856467974 2024-04-01T05:53:47.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:47.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285584 2024-04-01T05:53:47.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:47.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:53:47.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285584 2024-04-01T05:53:47.933 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:47.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285584 2024-04-01T05:53:47.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285584' 2024-04-01T05:53:47.934 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 356482285584 2024-04-01T05:53:47.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:53:48.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285584 -lt 356482285584 2024-04-01T05:53:48.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:53:48.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:48.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:53:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:53:48.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:53:48.864 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:53:48.865 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:48.865 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:53:49.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:53:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:53:49.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-04-01T05:53:51.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:53:51.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:53:51.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:53:51.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:53:51.888 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:53:51.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:53:51.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:53:51.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:53:51.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:53:51.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:53:51.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:53:51.915 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:51.915+0000 7f336fd21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:51.917 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:51.917+0000 7f336fd21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:51.918 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:51.919+0000 7f336fd21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:52.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:52.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:52.433+0000 7f336fd21640 -1 Falling back to public interface 2024-04-01T05:53:52.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:53.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:53.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:53.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:53:53.482 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:53.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:53.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:53.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:54.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:54.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:54.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:53:54.776 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:53:54.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:54.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:54.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:53:54.813+0000 7f336fd21640 -1 osd.0 120 log_to_monitors true 2024-04-01T05:53:55.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:53:56.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:53:56.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:53:56.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:53:56.164 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:53:56.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:53:56.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 125 up_thru 125 down_at 121 last_clean_interval [118,120) [v2:127.0.0.1:6802/3107607345,v1:127.0.0.1:6803/3107607345] [v2:127.0.0.1:6804/3107607345,v1:127.0.0.1:6805/3107607345] exists,up 222075b5-81e3-4066-81b4-c13bf2b80e2a 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:53:56.463 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:53:56.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:53:56.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:53:56.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:53:56.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:53:56.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:53:56.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:53:56.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:53:56.550 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:53:56.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:53:56.835 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:53:56.835 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T05:53:56.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:53:56.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:56.835 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:53:56.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=536870912002 2024-04-01T05:53:56.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 536870912002 2024-04-01T05:53:56.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-536870912002' 2024-04-01T05:53:56.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:56.940 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:53:57.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467977 2024-04-01T05:53:57.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467977 2024-04-01T05:53:57.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-536870912002 1-463856467977' 2024-04-01T05:53:57.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:53:57.046 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T05:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285587 2024-04-01T05:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285587 2024-04-01T05:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-536870912002 1-463856467977 2-356482285587' 2024-04-01T05:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-536870912002 2024-04-01T05:53:57.151 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:57.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:53:57.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-536870912002 2024-04-01T05:53:57.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=536870912002 2024-04-01T05:53:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 536870912002' 2024-04-01T05:53:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 536870912002 2024-04-01T05:53:57.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:57.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 536870912002 2024-04-01T05:53:57.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:53:58.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:53:58.438 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:53:58.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 536870912002 -lt 536870912002 2024-04-01T05:53:58.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:58.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467977 2024-04-01T05:53:58.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:58.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:53:58.816 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467977 2024-04-01T05:53:58.816 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:58.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467977 2024-04-01T05:53:58.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467977' 2024-04-01T05:53:58.817 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 463856467977 2024-04-01T05:53:58.818 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:53:59.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467977 -lt 463856467977 2024-04-01T05:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-356482285587 2024-04-01T05:53:59.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:53:59.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T05:53:59.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-356482285587 2024-04-01T05:53:59.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285587 2024-04-01T05:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 356482285587' 2024-04-01T05:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.2 seq 356482285587 2024-04-01T05:53:59.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T05:53:59.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285588 -lt 356482285587 2024-04-01T05:53:59.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:53:59.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:53:59.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:53:59.760 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:54:00.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:54:00.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:54:00.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:54:00.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:54:00.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:54:00.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:54:00.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:54:00.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-04-01T05:54:00.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-04-01T05:54:00.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T05:54:00.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-04-01T05:54:00.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-04-01T05:54:00.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:54:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-04-01T05:54:00.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-04-01T05:54:00.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-04-01T05:54:00.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-04-01T05:54:00.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:00.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:54:00.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:00.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:54:01.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:01.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-04-01T05:54:01.322 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-04-01T05:54:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T05:54:01.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:01.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:54:01.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:01.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:02.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:54:02.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:02.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:54:02.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:02.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:54:02.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:02.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:03.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:03.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:03.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:54:03.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:03.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:54:03.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:03.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:54:04.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:04.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:05.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:05.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:05.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:54:05.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:05.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:54:05.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:05.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:54:05.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:05.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:06.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:06.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:06.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T05:54:06.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:06.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T05:54:06.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:06.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T05:54:06.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:54:02.058626+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:06.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:54:06.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-04-01T05:54:06.826 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-04-01T05:54:06.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-04-01T05:54:06.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T05:54:06.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-04-01T05:54:06.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-04-01T05:54:06.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-04-01T05:54:06.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=125 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:54:06.860 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:54:06.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:54:06.884 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:54:06.885 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T05:54:06.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:54:06.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:54:06.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:54:06.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:54:06.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-04-01T05:54:06.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-04-01T05:54:06.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-04-01T05:54:06.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-04-01T05:54:06.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:54:06.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:06.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:54:06.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:06.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:54:07.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:07.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-04-01T05:54:07.455 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-04-01T05:54:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-04-01T05:51:34.085818+0000 last_deep_scrub_stamp 2024-04-01T05:54:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T05:54:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-04-01T05:54:07.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:54:07.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:07.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:54:07.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:07.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:54:07.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:07.468 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:54:07.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:07.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:08.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:54:08.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:08.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:54:08.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:08.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:54:09.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:09.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:10.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:10.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:10.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:54:10.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:10.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:54:10.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:10.034 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:54:10.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:10.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:11.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:11.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:11.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:54:11.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:11.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:54:11.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:11.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:54:11.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:51:34.085818+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:11.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:54:12.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:54:12.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:54:12.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:54:12.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:54:12.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:54:12.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:54:12.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:54:12.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:54:08.093257+0000 '>' 2024-04-01T05:51:34.085818+0000 2024-04-01T05:54:12.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:54:12.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-04-01T05:54:12.936 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-04-01T05:54:12.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-04-01T05:54:12.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T05:54:12.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-04-01T05:54:12.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-04-01T05:54:12.967 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-04-01T05:54:12.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=125 2024-04-01T05:54:12.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-04-01T05:54:12.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:54:12.970 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:54:12.971 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:54:12.972 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:54:12.972 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:54:12.972 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T05:54:12.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:54:12.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:54:12.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-04-01T05:54:12.994 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T05:54:12.994 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T05:54:12.994 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-04-01T05:54:12.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T05:54:13.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:54:13.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T05:54:13.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-04-01T05:54:13.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-04-01T05:54:13.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-04-01T05:54:13.415 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' removed 2024-04-01T05:54:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T05:54:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:54:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:54:13.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:54:13.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:54:13.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:54:13.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:54:13.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:54:13.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:54:13.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:54:13.536 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:54:13.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:54:13.537 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:54:13.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:54:13.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:54:13.539 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:54:13.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:54:13.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:54:13.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:54:13.540 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:54:13.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:54:13.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:54:13.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:54:13.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:54:13.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:13.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:13.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:54:13.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:54:13.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:54:13.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T05:54:13.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T05:54:13.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:54:13.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:54:13.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:54:13.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T05:54:13.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T05:54:13.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T05:54:13.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T05:54:13.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T05:54:13.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T05:54:13.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:54:13.575 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T05:54:13.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T05:54:13.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T05:54:13.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T05:54:13.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T05:54:13.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T05:54:13.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T05:54:13.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:13.579 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:13.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T05:54:13.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T05:54:13.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T05:54:13.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T05:54:13.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T05:54:13.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:13.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:13.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T05:54:13.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T05:54:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T05:54:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-04-01T05:54:13.965 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T05:54:13.965 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:54:13.965 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:54:13.966 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:54:13.966 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:13.966 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:13.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:54:13.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-04-01T05:54:14.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T05:54:14.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T05:54:14.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:54:14.150 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:54:14.151 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T05:54:14.151 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T05:54:14.151 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:54:14.151 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:54:14.151 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:54:14.152 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:54:14.152 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:14.152 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:14.152 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:54:14.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:54:14.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:54:14.236 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:14.237 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:14.237 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T05:54:14.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T05:54:14.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T05:54:14.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:15.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:54:15.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:54:15.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T05:54:15.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:54:15.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:54:15.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:54:15.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:54:15.180 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:54:15.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329' 2024-04-01T05:54:15.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:54:15.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHTApm+nirCxAAUAaZl/WHbuJ5VWaYPMexKg== 2024-04-01T05:54:15.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHTApm+nirCxAAUAaZl/WHbuJ5VWaYPMexKg=="}' 2024-04-01T05:54:15.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 -i td/osd-scrub-repair/0/new.json 2024-04-01T05:54:15.399 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:54:15.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T05:54:15.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHTApm+nirCxAAUAaZl/WHbuJ5VWaYPMexKg== --osd-uuid 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:54:15.438 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.437+0000 7f05f4373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:15.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.449+0000 7f05f4373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:15.451 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.450+0000 7f05f4373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:15.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.451+0000 7f05f4373640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:54:15.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.451+0000 7f05f4373640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:54:15.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.451+0000 7f05f4373640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T05:54:15.542 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:15.541+0000 7f05f4373640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T05:54:21.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T05:54:21.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:54:21.168 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T05:54:21.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T05:54:21.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:54:21.578 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T05:54:21.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T05:54:21.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:54:21.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:54:21.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:54:21.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:54:21.604 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:21.603+0000 7fb90aa1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:21.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:21.605+0000 7fb90aa1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:21.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:21.606+0000 7fb90aa1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:21.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T05:54:21.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:54:21.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:54:21.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:54:21.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:54:21.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:21.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:54:21.866 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:54:21.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:21.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:22.122 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:22.121+0000 7fb90aa1e640 -1 Falling back to public interface 2024-04-01T05:54:22.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:23.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:23.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:23.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:54:23.150 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:54:23.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:23.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:23.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:24.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:24.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:24.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:54:24.443 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:54:24.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:24.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:24.468 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:24.467+0000 7fb90aa1e640 -1 osd.0 0 log_to_monitors true 2024-04-01T05:54:24.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:25.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:25.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:25.762 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:54:25.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:54:25.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:25.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:26.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:27.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:27.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:27.067 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:54:27.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:54:27.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:27.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:27.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:28.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:28.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:28.364 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:54:28.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:54:28.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:28.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:28.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:29.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:29.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:29.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:54:29.661 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:54:29.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:29.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:29.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:30.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:30.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:30.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:54:30.954 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:54:30.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:30.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:31.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:32.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:32.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:32.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:54:32.275 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:54:32.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:32.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:32.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:33.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:33.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:33.570 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:54:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:54:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:33.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:34.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:34.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:34.863 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:54:34.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:54:34.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:34.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:35.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:36.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:36.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:36.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:54:36.244 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:54:36.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:36.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:37.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:38.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:38.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:38.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:54:38.222 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:54:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:38.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:38.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:39.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:39.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:39.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:54:39.599 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:54:39.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:39.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:40.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:41.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:41.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:41.402 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:54:41.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:54:41.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:41.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:41.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:42.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:42.821 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:54:42.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:42.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:54:42.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:42.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:43.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:44.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:44.541 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:54:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:54:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:44.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:45.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:45.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:45.919 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:54:45.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:54:45.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:45.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:46.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:47.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:47.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:47.747 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:54:47.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:54:47.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:47.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:54:48.039 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3336298103,v1:127.0.0.1:6803/3336298103] [v2:127.0.0.1:6804/3336298103,v1:127.0.0.1:6805/3336298103] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T05:54:48.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:54:48.044 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T05:54:48.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T05:54:48.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:54:48.046 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T05:54:48.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:54:48.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6283fff3-1f5c-405a-aac0-cd967417ba99' 2024-04-01T05:54:48.046 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:54:48.046 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T05:54:48.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoTApmuCOTAxAAImJkgUL8VTM89U8ZV+/Tmw== 2024-04-01T05:54:48.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoTApmuCOTAxAAImJkgUL8VTM89U8ZV+/Tmw=="}' 2024-04-01T05:54:48.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6283fff3-1f5c-405a-aac0-cd967417ba99 -i td/osd-scrub-repair/1/new.json 2024-04-01T05:54:48.433 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:54:48.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T05:54:48.444 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoTApmuCOTAxAAImJkgUL8VTM89U8ZV+/Tmw== --osd-uuid 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:54:48.473 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.473+0000 7f67eed81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:48.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.509+0000 7f67eed81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:48.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.510+0000 7f67eed81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:48.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.510+0000 7f67eed81640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:54:48.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.510+0000 7f67eed81640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:54:48.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.510+0000 7f67eed81640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T05:54:48.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:48.584+0000 7f67eed81640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T05:54:54.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T05:54:54.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T05:54:54.856 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T05:54:54.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T05:54:54.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T05:54:55.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T05:54:55.223 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T05:54:55.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:54:55.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T05:54:55.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T05:54:55.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T05:54:55.249 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:55.249+0000 7fa2d2c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:55.250 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:55.251+0000 7fa2d2c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:55.251 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:55.252+0000 7fa2d2c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:54:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T05:54:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:54:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:54:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:54:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:54:55.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:55.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:54:55.512 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T05:54:55.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:55.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:54:55.767 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:55.767+0000 7fa2d2c13640 -1 Falling back to public interface 2024-04-01T05:54:55.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:56.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:56.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:56.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:54:56.819 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T05:54:56.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:56.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:54:57.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:58.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:54:58.092+0000 7fa2d2c13640 -1 osd.1 0 log_to_monitors true 2024-04-01T05:54:58.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:58.111 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T05:54:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:54:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:54:58.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:54:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:54:59.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:54:59.435 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T05:54:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:54:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:54:59.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:54:59.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:00.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:00.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:00.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T05:55:00.731 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T05:55:00.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:00.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:01.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:02.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:02.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:02.142 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T05:55:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T05:55:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:02.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:02.527 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:03.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:03.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:03.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T05:55:03.529 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T05:55:03.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:03.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:03.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:04.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T05:55:04.892 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T05:55:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:04.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:05.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:06.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:06.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:06.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T05:55:06.191 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T05:55:06.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:06.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:06.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:07.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:07.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:07.501 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T05:55:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T05:55:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:07.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:07.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:08.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:08.884 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T05:55:08.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:08.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T05:55:08.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:08.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:09.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:10.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:10.179 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T05:55:10.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T05:55:10.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:10.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:10.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:11.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:11.925 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T05:55:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T05:55:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:11.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:12.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:13.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:13.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:13.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T05:55:13.358 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T05:55:13.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:13.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:13.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:14.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:14.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:14.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T05:55:14.654 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T05:55:14.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:14.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:15.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:16.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:16.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:16.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T05:55:16.211 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T05:55:16.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:16.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:17.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:18.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:18.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:18.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T05:55:18.003 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T05:55:18.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:18.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:18.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:19.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:19.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:19.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T05:55:19.623 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T05:55:19.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:19.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:20.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:55:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:55:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:55:21.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T05:55:21.057 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T05:55:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:55:21.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:55:21.468 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2485468463,v1:127.0.0.1:6811/2485468463] [v2:127.0.0.1:6812/2485468463,v1:127.0.0.1:6813/2485468463] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:55:21.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:55:21.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:55:21.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:55:21.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-04-01T05:55:21.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T05:55:21.786 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T05:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T05:55:21.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T05:55:22.176 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T05:55:22.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:55:23.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T05:55:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-04-01T05:55:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:55:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:55:24.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:55:24.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:55:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:55:24.529 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:55:24.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:55:24.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:55:24.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:55:24.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:55:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:55:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:55:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:55:24.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:55:24.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:55:24.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:55:24.909 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:55:24.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:55:24.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:55:24.909 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:55:25.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-04-01T05:55:25.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-04-01T05:55:25.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-04-01T05:55:25.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:55:25.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:55:25.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-04-01T05:55:25.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-04-01T05:55:25.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672964' 2024-04-01T05:55:25.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:55:25.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-04-01T05:55:25.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:55:25.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:55:25.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-04-01T05:55:25.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:55:25.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-04-01T05:55:25.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-04-01T05:55:25.123 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836492 2024-04-01T05:55:25.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:55:25.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-04-01T05:55:25.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:55:26.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:55:26.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:55:26.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-04-01T05:55:26.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:55:26.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-04-01T05:55:26.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:55:26.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:55:26.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-04-01T05:55:26.724 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:55:26.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-04-01T05:55:26.726 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672964 2024-04-01T05:55:26.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-04-01T05:55:26.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:55:27.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-04-01T05:55:27.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:55:27.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:55:27.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:55:27.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T05:55:27.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:55:27.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:55:27.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:55:27.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:55:27.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:55:27.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:55:27.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:55:27.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T05:55:27.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:55:27.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:55:27.675 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:55:28.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T05:55:28.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:55:28.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:55:28.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-04-01T05:55:28.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-04-01T05:55:28.520 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' created 2024-04-01T05:55:28.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:55:29.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-04-01T05:55:29.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-04-01T05:55:30.012 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'csr_pool' created 2024-04-01T05:55:30.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T05:55:31.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-04-01T05:55:31.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:55:31.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:55:31.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:55:31.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:55:31.026 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:55:31.026 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:55:31.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:55:31.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:55:31.026 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:55:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:55:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:55:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:55:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:55:31.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:55:31.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:55:31.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:55:31.437 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:55:31.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:55:31.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:55:31.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:55:31.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-04-01T05:55:31.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-04-01T05:55:31.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-04-01T05:55:31.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:55:31.543 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:55:31.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-04-01T05:55:31.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-04-01T05:55:31.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672967' 2024-04-01T05:55:31.649 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:55:31.649 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-04-01T05:55:31.649 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:55:31.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:55:31.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-04-01T05:55:31.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:55:31.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-04-01T05:55:31.652 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836494 2024-04-01T05:55:31.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-04-01T05:55:31.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:55:31.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-04-01T05:55:31.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:55:32.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:55:32.935 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:55:33.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-04-01T05:55:33.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:55:34.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:55:34.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:55:34.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836494 2024-04-01T05:55:34.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:55:34.528 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-04-01T05:55:34.528 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:55:34.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:55:34.530 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-04-01T05:55:34.530 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:55:34.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-04-01T05:55:34.531 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672967 2024-04-01T05:55:34.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-04-01T05:55:34.532 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:55:34.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-04-01T05:55:34.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:55:34.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:55:34.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:55:35.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:55:35.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:55:35.205 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:55:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:55:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:55:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:55:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:55:35.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:55:35.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:55:35.485 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:55:35.485 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:55:35.485 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:55:35.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:55:35.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:55:35.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:55:35.835 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-04-01T05:55:35.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:35.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-04-01T05:55:35.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-04-01T05:55:35.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:35.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:35.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-04-01T05:55:35.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:35.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:35.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:36.216 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:36.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:36.760 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:36.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:36.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:36.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:36.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-04-01T05:55:37.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:37.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:37.697 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:37.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:38.091 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:38.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:38.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:38.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:38.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-04-01T05:55:38.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-04-01T05:55:38.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:38.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-04-01T05:55:38.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-04-01T05:55:38.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:38.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:38.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-04-01T05:55:38.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:38.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:38.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:38.986 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:38.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:39.513 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:39.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:39.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:39.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:39.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-04-01T05:55:39.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:39.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:40.383 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:40.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:40.793 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:40.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:40.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:40.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:41.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-04-01T05:55:41.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:41.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:41.739 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:41.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:42.275 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:42.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:42.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:42.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:42.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-04-01T05:55:42.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:42.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:43.153 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:43.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:43.572 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:43.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:43.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:43.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:43.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-04-01T05:55:44.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-04-01T05:55:44.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:44.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:44.543 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:44.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:45.055 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:45.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:45.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:45.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:45.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-04-01T05:55:45.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-04-01T05:55:45.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:45.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:46.007 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:46.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:46.551 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:46.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:46.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:46.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:46.797 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-04-01T05:55:46.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-04-01T05:55:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-04-01T05:55:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-04-01T05:55:47.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:47.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:47.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-04-01T05:55:47.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:47.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:47.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:47.429 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:47.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:47.966 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:47.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:47.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:48.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-04-01T05:55:48.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-04-01T05:55:48.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:48.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-04-01T05:55:48.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-04-01T05:55:48.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:48.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:48.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-04-01T05:55:48.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:48.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:48.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:48.844 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:48.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:49.372 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:49.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:49.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:49.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:49.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-04-01T05:55:49.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-04-01T05:55:49.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:49.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:50.384 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:50.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:50.802 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:50.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:50.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:50.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:51.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-04-01T05:55:51.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-04-01T05:55:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-04-01T05:55:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-04-01T05:55:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:51.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-04-01T05:55:51.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:51.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:51.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:51.764 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:51.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:52.283 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:52.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:52.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:52.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:52.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-04-01T05:55:52.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-04-01T05:55:52.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:52.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-04-01T05:55:52.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-04-01T05:55:52.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:52.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:52.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-04-01T05:55:52.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:52.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:52.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:53.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:53.765 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:53.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:53.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:53.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:54.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-04-01T05:55:54.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:54.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:54.677 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:54.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:55.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-04-01T05:55:55.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-04-01T05:55:55.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:55.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:56.099 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:56.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:56.677 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:56.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:56.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:56.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:56.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-04-01T05:55:57.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-04-01T05:55:57.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:57.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:57.555 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:57.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:58.069 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:58.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:58.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-04-01T05:55:58.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-04-01T05:55:58.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:55:58.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:55:58.961 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:55:58.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:55:59.482 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:55:59.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:55:59.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:55:59.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:55:59.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-04-01T05:55:59.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-04-01T05:56:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:56:00.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:56:00.384 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:56:00.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:56:00.786 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:56:00.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:56:00.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:56:00.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:56:01.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-04-01T05:56:01.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-04-01T05:56:01.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:01.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-04-01T05:56:01.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-04-01T05:56:01.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T05:56:01.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T05:56:01.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-04-01T05:56:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T05:56:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T05:56:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T05:56:01.782 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T05:56:01.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T05:56:02.176 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T05:56:02.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T05:56:02.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T05:56:02.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-04-01T05:56:02.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-04-01T05:56:02.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-04-01T05:56:02.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-04-01T05:56:02.793 INFO:tasks.workunit.client.0.smithi067.stderr:1025+0 records in 2024-04-01T05:56:02.793 INFO:tasks.workunit.client.0.smithi067.stderr:1025+0 records out 2024-04-01T05:56:02.793 INFO:tasks.workunit.client.0.smithi067.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.0018447 s, 569 MB/s 2024-04-01T05:56:02.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-04-01T05:56:02.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-04-01T05:56:02.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-04-01T05:56:02.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-04-01T05:56:02.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-04-01T05:56:02.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-04-01T05:56:02.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T05:56:03.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-04-01T05:56:03.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-04-01T05:56:03.261 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-04-01T05:56:03.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-04-01T05:56:03.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-04-01T05:56:03.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T05:56:03.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-04-01T05:56:03.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.70255/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-04-01T05:56:03.626 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T05:56:03.626 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-04-01T05:56:03.626 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T05:56:03.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-04-01T05:56:03.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-04-01T05:56:03.714 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T05:56:03.715 INFO:tasks.workunit.client.0.smithi067.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-04-01T05:56:03.715 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T05:56:03.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-04-01T05:56:03.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-04-01T05:56:03.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:56:03.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:56:03.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:56:03.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:56:03.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:56:04.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-04-01T05:55:29.895026+0000 2024-04-01T05:56:04.003 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-04-01T05:56:04.270 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-04-01T05:56:04.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-04-01T05:55:29.895026+0000 last_deep_scrub_stamp 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:55:29.895026+0000 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:56:04.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:56:04.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:55:29.895026+0000 '>' 2024-04-01T05:55:29.895026+0000 2024-04-01T05:56:04.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:56:05.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:56:05.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:56:05.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:56:05.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:56:05.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:56:05.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:56:05.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:56:05.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:55:29.895026+0000 '>' 2024-04-01T05:55:29.895026+0000 2024-04-01T05:56:05.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:56:06.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:56:06.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:56:06.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:56:06.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:56:06.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:56:06.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:56:06.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:56:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:55:29.895026+0000 '>' 2024-04-01T05:55:29.895026+0000 2024-04-01T05:56:07.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T05:56:08.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T05:56:08.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T05:56:08.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T05:56:08.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T05:56:08.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T05:56:08.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T05:56:08.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T05:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:56:05.193652+0000 '>' 2024-04-01T05:55:29.895026+0000 2024-04-01T05:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T05:56:08.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-04-01T05:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-04-01T05:56:08.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:56:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:56:08.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:56:08.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:10.196 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-04-01T05:56:10.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:56:10.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:56:10.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:56:10.745 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:56:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:56:10.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:56:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:56:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:56:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:56:10.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:56:10.772 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:10.772+0000 7f4bd5526640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:10.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:10.775+0000 7f4bd5526640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:10.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:10.776+0000 7f4bd5526640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:11.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:56:11.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:56:11.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:56:11.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:56:11.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:56:11.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:11.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:56:11.039 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:56:11.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:11.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:11.294 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:11.294+0000 7f4bd5526640 -1 Falling back to public interface 2024-04-01T05:56:11.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:56:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:56:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:12.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:12.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:13.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:13.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:13.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:56:13.624 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:56:13.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:13.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:13.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:13.831+0000 7f4bd5526640 -1 osd.1 61 log_to_monitors true 2024-04-01T05:56:13.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:14.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:14.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:14.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:56:14.932 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:56:14.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:14.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:15.223 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [10,61) [v2:127.0.0.1:6810/4202328467,v1:127.0.0.1:6811/4202328467] [v2:127.0.0.1:6812/4202328467,v1:127.0.0.1:6813/4202328467] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:56:15.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:56:15.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:56:15.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:56:15.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:56:15.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:56:15.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:56:15.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:56:15.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:56:15.669 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:56:15.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:56:15.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:15.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:56:15.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-04-01T05:56:15.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-04-01T05:56:15.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-04-01T05:56:15.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:15.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:56:15.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-04-01T05:56:15.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-04-01T05:56:15.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-279172874242' 2024-04-01T05:56:15.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:15.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-04-01T05:56:15.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:15.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:56:15.914 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-04-01T05:56:15.914 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:15.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-04-01T05:56:15.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-01T05:56:15.915 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836504 2024-04-01T05:56:15.916 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:16.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-04-01T05:56:16.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:56:17.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:56:17.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:17.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-01T05:56:17.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:17.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-04-01T05:56:17.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:17.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:56:17.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-04-01T05:56:17.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:17.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-04-01T05:56:17.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-04-01T05:56:17.507 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 279172874242 2024-04-01T05:56:17.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:56:17.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874242 -lt 279172874242 2024-04-01T05:56:17.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:56:17.790 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:17.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:56:18.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:56:18.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:56:18.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:18.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:18.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:56:18.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:56:18.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:56:18.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:18.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-04-01T05:56:18.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:56:18.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:56:18.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:56:18.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:56:18.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:56:19.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:56:19.086 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:56:21.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:21.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:56:21.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:56:21.052 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:56:21.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:56:21.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:56:21.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:56:21.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:56:21.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:21.078+0000 7f24f360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:21.080 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:21.080+0000 7f24f360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:21.081 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:21.082+0000 7f24f360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:21.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:56:21.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:56:21.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:56:21.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:56:21.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:56:21.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:21.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:56:21.347 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:56:21.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:21.347 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:21.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:21.594+0000 7f24f360d640 -1 Falling back to public interface 2024-04-01T05:56:21.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:22.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:22.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:22.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:56:22.628 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:56:22.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:22.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:22.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:56:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:56:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:23.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:23.945 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:23.945+0000 7f24f360d640 -1 osd.0 67 log_to_monitors true 2024-04-01T05:56:24.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:25.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:25.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:25.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:56:25.317 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:56:25.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:25.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:25.612 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [5,67) [v2:127.0.0.1:6802/3663656713,v1:127.0.0.1:6803/3663656713] [v2:127.0.0.1:6804/3663656713,v1:127.0.0.1:6805/3663656713] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:56:25.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:56:25.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:56:25.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:56:25.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:56:25.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:56:25.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:56:25.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:56:25.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:56:25.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:56:25.986 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:56:25.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:56:25.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:25.987 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:56:26.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-04-01T05:56:26.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-04-01T05:56:26.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678018' 2024-04-01T05:56:26.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:26.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:56:26.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874245 2024-04-01T05:56:26.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874245 2024-04-01T05:56:26.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678018 1-279172874245' 2024-04-01T05:56:26.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:26.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-304942678018 2024-04-01T05:56:26.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:26.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:56:26.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-304942678018 2024-04-01T05:56:26.199 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:26.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-04-01T05:56:26.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 304942678018' 2024-04-01T05:56:26.200 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 304942678018 2024-04-01T05:56:26.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 304942678018 2024-04-01T05:56:26.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:56:27.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:56:27.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:27.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-04-01T05:56:27.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:27.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874245 2024-04-01T05:56:27.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:27.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:56:27.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874245 2024-04-01T05:56:27.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:27.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874245 2024-04-01T05:56:27.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874245' 2024-04-01T05:56:27.785 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 279172874245 2024-04-01T05:56:27.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:56:28.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874245 -lt 279172874245 2024-04-01T05:56:28.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:56:28.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:28.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:56:28.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:56:28.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:56:28.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:56:28.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:28.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:29.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:56:29.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:56:29.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:56:29.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:29.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-04-01T05:56:29.090 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-04-01T05:56:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:56:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:56:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-04-01T05:56:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:56:29.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:56:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:56:29.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:56:29.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-04-01T05:56:30.651 INFO:tasks.workunit.client.0.smithi067.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-04-01T05:56:31.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:56:31.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:56:31.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:56:31.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:56:31.175 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:56:31.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:56:31.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:56:31.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:56:31.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:56:31.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:56:31.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:56:31.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:31.201+0000 7f4ce0404640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:31.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:31.203+0000 7f4ce0404640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:31.205 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:31.204+0000 7f4ce0404640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:56:31.468 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:56:31.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:31.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:31.724 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:31.723+0000 7f4ce0404640 -1 Falling back to public interface 2024-04-01T05:56:31.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:32.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:32.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:32.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:56:32.750 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:56:32.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:32.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:33.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:34.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:34.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:34.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:56:34.043 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:56:34.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:34.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:34.084 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:34.082+0000 7f4ce0404640 -1 osd.1 73 log_to_monitors true 2024-04-01T05:56:34.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:35.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:35.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:35.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:56:35.412 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:56:35.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:35.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:35.708 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [65,73) [v2:127.0.0.1:6810/2889893135,v1:127.0.0.1:6811/2889893135] [v2:127.0.0.1:6812/2889893135,v1:127.0.0.1:6813/2889893135] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:56:35.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:56:35.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:56:35.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:56:35.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:56:35.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:56:35.709 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:56:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:56:35.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:56:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:56:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:56:35.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:56:35.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:56:36.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:56:36.074 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:56:36.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:56:36.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:36.075 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:56:36.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678021 2024-04-01T05:56:36.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678021 2024-04-01T05:56:36.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678021' 2024-04-01T05:56:36.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:36.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:56:36.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-04-01T05:56:36.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-04-01T05:56:36.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-304942678021 1-330712481794' 2024-04-01T05:56:36.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:36.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-304942678021 2024-04-01T05:56:36.286 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:36.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:56:36.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-304942678021 2024-04-01T05:56:36.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:36.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678021 2024-04-01T05:56:36.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 304942678021' 2024-04-01T05:56:36.288 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 304942678021 2024-04-01T05:56:36.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:36.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678020 -lt 304942678021 2024-04-01T05:56:36.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:56:37.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:56:37.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678021 -lt 304942678021 2024-04-01T05:56:37.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:37.868 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-04-01T05:56:37.868 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:37.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:56:37.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-04-01T05:56:37.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:37.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-04-01T05:56:37.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-04-01T05:56:37.871 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 330712481794 2024-04-01T05:56:37.872 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:56:38.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-04-01T05:56:38.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:56:38.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:38.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:38.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:56:38.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:56:38.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:56:38.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:56:38.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:56:38.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:56:38.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:56:38.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:56:38.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:56:38.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:56:38.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:38.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:39.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:56:39.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:56:39.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:56:39.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:39.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-04-01T05:56:39.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-04-01T05:56:39.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:56:39.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:56:39.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:39.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:56:39.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:56:39.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:56:39.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:56:39.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:39.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:56:39.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:56:39.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:56:39.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:56:39.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:56:39.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:56:39.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:56:39.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:56:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:56:39.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:56:39.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:56:39.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:56:41.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:41.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:56:41.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:56:41.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:56:41.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:41.298+0000 7f138741f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:41.300 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:41.300+0000 7f138741f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:41.301 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:41.301+0000 7f138741f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:41.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:56:41.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:56:41.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:56:41.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:56:41.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:56:41.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:41.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:56:41.564 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:56:41.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:41.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:41.816 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:41.816+0000 7f138741f640 -1 Falling back to public interface 2024-04-01T05:56:41.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:56:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:56:42.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:42.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:43.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:44.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:44.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:44.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:56:44.140 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:56:44.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:44.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:44.208 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:44.208+0000 7f138741f640 -1 osd.0 78 log_to_monitors true 2024-04-01T05:56:44.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:56:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:56:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [71,78) [v2:127.0.0.1:6802/733229517,v1:127.0.0.1:6803/733229517] [v2:127.0.0.1:6804/733229517,v1:127.0.0.1:6805/733229517] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:56:45.795 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:56:45.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:56:45.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:56:45.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:56:45.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:56:45.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:56:45.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:56:45.883 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:56:46.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:56:46.165 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:56:46.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:56:46.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:46.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:56:46.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-04-01T05:56:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-04-01T05:56:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-04-01T05:56:46.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:46.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:56:46.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-04-01T05:56:46.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-04-01T05:56:46.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-04-01T05:56:46.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:46.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-04-01T05:56:46.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:46.377 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:56:46.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-04-01T05:56:46.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:46.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-04-01T05:56:46.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-04-01T05:56:46.379 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 352187318274 2024-04-01T05:56:46.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:46.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-04-01T05:56:46.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:56:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:56:47.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:47.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-04-01T05:56:47.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:47.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-04-01T05:56:47.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:47.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:56:47.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-04-01T05:56:47.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:47.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-04-01T05:56:47.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-04-01T05:56:47.960 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 330712481797 2024-04-01T05:56:47.960 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:56:48.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-04-01T05:56:48.242 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:56:48.242 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:48.242 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:56:48.620 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:56:48.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:56:48.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:56:48.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:48.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:49.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-04-01T05:56:49.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:56:49.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:56:49.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-04-01T05:56:51.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:56:51.349 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:56:51.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:56:51.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:56:51.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:56:51.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:56:51.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:56:51.379 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:51.379+0000 7f923002a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:51.382 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:51.382+0000 7f923002a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:51.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:51.383+0000 7f923002a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:56:51.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:51.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:51.899 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:51.899+0000 7f923002a640 -1 Falling back to public interface 2024-04-01T05:56:51.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:56:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:56:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:52.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:53.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:54.216 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:56:54.216+0000 7f923002a640 -1 osd.1 84 log_to_monitors true 2024-04-01T05:56:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:56:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:56:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:54.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:54.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:56:55.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:56:55.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:56:55.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:56:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:56:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:56:55.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:56:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/4210018390,v1:127.0.0.1:6811/4210018390] [v2:127.0.0.1:6812/4210018390,v1:127.0.0.1:6813/4210018390] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:56:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:56:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:56:55.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:56:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:56:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:56:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:56:55.838 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:56:55.838 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:56:55.839 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:56:55.839 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:56:55.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:56:55.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:56:55.839 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:56:55.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:56:55.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:56:55.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:56:55.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:56:55.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:56:55.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:56:56.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:56:56.209 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:56:56.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:56:56.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:56.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:56:56.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-04-01T05:56:56.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-04-01T05:56:56.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-04-01T05:56:56.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:56:56.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:56:56.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-04-01T05:56:56.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-04-01T05:56:56.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-04-01T05:56:56.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:56.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-04-01T05:56:56.420 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:56.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:56:56.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-04-01T05:56:56.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:56.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-04-01T05:56:56.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-04-01T05:56:56.422 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 352187318277 2024-04-01T05:56:56.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:56.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-04-01T05:56:56.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:56:57.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:56:57.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:56:57.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318277 -lt 352187318277 2024-04-01T05:56:57.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:56:57.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-04-01T05:56:57.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:56:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:56:58.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-04-01T05:56:58.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:56:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-04-01T05:56:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-04-01T05:56:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 377957122050 2024-04-01T05:56:58.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:56:58.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-04-01T05:56:58.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:56:58.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:58.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:58.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:56:58.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:56:58.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:56:58.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:56:58.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:56:58.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:56:58.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:56:58.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:56:58.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:56:58.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:56:58.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:56:58.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:56:59.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:56:59.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:56:59.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:56:59.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:56:59.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-04-01T05:56:59.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-04-01T05:56:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:56:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:56:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-04-01T05:56:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-04-01T05:56:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:56:59.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:56:59.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-04-01T05:57:01.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:57:01.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:57:01.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:57:01.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:57:01.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:57:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:57:01.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:57:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:57:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:57:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:57:01.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:57:01.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:57:01.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:57:01.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:57:01.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:57:01.277 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:01.276+0000 7f8e51a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:01.279 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:01.279+0000 7f8e51a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:01.280 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:01.280+0000 7f8e51a18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:01.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:57:01.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:01.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:01.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:01.802+0000 7f8e51a18640 -1 Falling back to public interface 2024-04-01T05:57:01.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:02.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:02.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:02.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:57:02.865 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:57:02.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:02.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:03.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:04.111 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:04.111+0000 7f8e51a18640 -1 osd.0 89 log_to_monitors true 2024-04-01T05:57:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:57:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:57:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:04.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:04.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:05.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:05.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:05.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:57:05.590 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:57:05.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:05.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:05.882 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/3356233683,v1:127.0.0.1:6803/3356233683] [v2:127.0.0.1:6804/3356233683,v1:127.0.0.1:6805/3356233683] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:57:05.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:57:05.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:57:05.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:57:05.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:57:05.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:57:05.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:57:05.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:57:05.888 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:57:05.888 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:57:05.888 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:57:05.888 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:57:05.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:57:05.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:57:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:57:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:57:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:57:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:57:05.970 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:57:05.970 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:57:06.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:57:06.253 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:57:06.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:57:06.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:06.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:57:06.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-04-01T05:57:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-04-01T05:57:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-04-01T05:57:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:06.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:57:06.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-04-01T05:57:06.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-04-01T05:57:06.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-04-01T05:57:06.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:06.477 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-04-01T05:57:06.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:06.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:57:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-04-01T05:57:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-04-01T05:57:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-04-01T05:57:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 399431958530 2024-04-01T05:57:06.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:06.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-04-01T05:57:06.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:57:07.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:57:07.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:08.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-04-01T05:57:08.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:08.058 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-04-01T05:57:08.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:08.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:57:08.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-04-01T05:57:08.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:08.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-04-01T05:57:08.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-04-01T05:57:08.062 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 377957122053 2024-04-01T05:57:08.062 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:57:08.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122053 -lt 377957122053 2024-04-01T05:57:08.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:57:08.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:08.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:57:08.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:08.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:08.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:08.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:08.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:08.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:08.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:08.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:57:08.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:08.998 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:08.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:09.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:57:09.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:57:09.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:57:09.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-04-01T05:57:09.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:57:09.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-04-01T05:57:09.354 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:57:09.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:57:09.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:57:09.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:57:09.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:57:09.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:57:09.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:57:09.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:57:09.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:09.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-04-01T05:57:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:57:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:57:11.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:57:11.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:57:11.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:57:11.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:57:11.301 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:57:11.301 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:57:11.302 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:57:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:57:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:57:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:57:11.302 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:57:11.328 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:11.327+0000 7f5d90268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:11.330 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:11.329+0000 7f5d90268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:11.332 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:11.331+0000 7f5d90268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:11.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:57:11.597 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:57:11.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:11.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:11.851 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:11.850+0000 7f5d90268640 -1 Falling back to public interface 2024-04-01T05:57:11.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:12.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:12.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:12.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:57:12.882 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:57:12.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:12.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:13.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:14.137 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:14.136+0000 7f5d90268640 -1 osd.1 95 log_to_monitors true 2024-04-01T05:57:14.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:14.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:14.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:57:14.176 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:57:14.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:14.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 99 up_thru 94 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/2091673608,v1:127.0.0.1:6811/2091673608] [v2:127.0.0.1:6812/2091673608,v1:127.0.0.1:6813/2091673608] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:57:14.558 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:57:14.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:57:14.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:57:14.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:57:14.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:57:14.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:57:14.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:57:14.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:57:14.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:57:14.646 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:57:14.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:57:14.956 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:57:14.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:57:14.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:14.956 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:57:15.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-04-01T05:57:15.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-04-01T05:57:15.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-04-01T05:57:15.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:15.143 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:57:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-04-01T05:57:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-04-01T05:57:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-04-01T05:57:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:15.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-04-01T05:57:15.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:15.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:57:15.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-04-01T05:57:15.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-04-01T05:57:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-04-01T05:57:15.330 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 399431958533 2024-04-01T05:57:15.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958533 -lt 399431958533 2024-04-01T05:57:15.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:15.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-04-01T05:57:15.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:15.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:57:15.775 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-04-01T05:57:15.775 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-04-01T05:57:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-04-01T05:57:15.776 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 425201762306 2024-04-01T05:57:15.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:57:16.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-04-01T05:57:16.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:57:16.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:16.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:16.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:16.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:16.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:57:16.697 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:16.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:16.698 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:17.046 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:57:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:57:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:57:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:57:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:57:17.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:17.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:17.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:17.427 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:57:17.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:17.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:17.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:57:17.796 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:57:18.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:57:18.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:57:18.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:57:18.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:57:18.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:18.348 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:18.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:57:18.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:18.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:18.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:57:18.990 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:57:19.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:57:19.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:57:19.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-01T05:57:19.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:57:19.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-01T05:57:19.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:57:19.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:19.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:19.759 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:19.759 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:19.759 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:19.759 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:19.759 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:20.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:57:20.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:20.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:20.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:20.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:57:20.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:57:20.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:57:20.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:57:20.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-01T05:57:20.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:57:20.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-01T05:57:21.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:57:21.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:21.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:21.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:21.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:21.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:21.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:21.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:21.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:57:21.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:21.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:21.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:22.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:57:22.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:57:22.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:57:22.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-04-01T05:57:22.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:57:22.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-04-01T05:57:22.233 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-04-01T05:57:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:57:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:57:22.233 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-04-01T05:57:22.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-04-01T05:57:22.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-04-01T05:57:22.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-04-01T05:57:22.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:57:22.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:57:22.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-04-01T05:57:24.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:57:24.399 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:57:24.400 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:57:24.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:57:24.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:57:24.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:57:24.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:57:24.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:57:24.402 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:57:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:57:24.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:57:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:57:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:57:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:57:24.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:57:24.429 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:24.429+0000 7f1c75a8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:24.431 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:24.431+0000 7f1c75a8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:24.433 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:24.433+0000 7f1c75a8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:57:24.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:24.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:24.951 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:24.951+0000 7f1c75a8b640 -1 Falling back to public interface 2024-04-01T05:57:24.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:25.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:25.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:25.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:57:25.982 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:57:25.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:25.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:26.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:27.237 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:27.237+0000 7f1c75a8b640 -1 osd.0 101 log_to_monitors true 2024-04-01T05:57:27.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:27.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:27.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:57:27.286 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:57:27.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:27.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:27.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:28.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:28.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/913023947,v1:127.0.0.1:6803/913023947] [v2:127.0.0.1:6804/913023947,v1:127.0.0.1:6805/913023947] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:57:28.929 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:57:28.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:57:28.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:57:28.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:57:29.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:57:29.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:57:29.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:57:29.298 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:57:29.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:57:29.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:29.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:57:29.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-04-01T05:57:29.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-04-01T05:57:29.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-04-01T05:57:29.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:29.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:57:29.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762310 2024-04-01T05:57:29.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762310 2024-04-01T05:57:29.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762310' 2024-04-01T05:57:29.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:29.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-04-01T05:57:29.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:29.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:57:29.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-04-01T05:57:29.510 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:29.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-04-01T05:57:29.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-04-01T05:57:29.511 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 450971566082 2024-04-01T05:57:29.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:29.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-04-01T05:57:29.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:57:30.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:57:30.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:31.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-04-01T05:57:31.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:57:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:57:32.096 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:32.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-04-01T05:57:32.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:32.390 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762310 2024-04-01T05:57:32.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:32.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:57:32.392 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762310 2024-04-01T05:57:32.392 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:32.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762310 2024-04-01T05:57:32.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762310' 2024-04-01T05:57:32.393 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 425201762310 2024-04-01T05:57:32.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:57:32.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762310 -lt 425201762310 2024-04-01T05:57:32.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:57:32.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:32.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:33.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:57:33.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:33.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:33.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:33.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:33.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:33.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:33.092 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:33.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:57:33.370 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:33.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:33.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:57:33.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:57:33.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:57:34.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:34.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-04-01T05:57:35.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:57:35.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:57:35.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:57:35.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:57:35.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:57:35.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:57:35.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:57:35.784 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:57:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:57:35.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:57:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:57:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:57:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:57:35.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:57:35.811 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:35.811+0000 7fe0915ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:35.813 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:35.813+0000 7fe0915ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:35.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:35.815+0000 7fe0915ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:36.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:36.329 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:36.329+0000 7fe0915ac640 -1 Falling back to public interface 2024-04-01T05:57:36.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:57:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:57:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:37.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:37.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:38.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:38.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:38.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:57:38.700 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:57:38.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:38.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:38.779 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:38.778+0000 7fe0915ac640 -1 osd.1 106 log_to_monitors true 2024-04-01T05:57:39.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:40.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:40.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:40.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:57:40.105 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:57:40.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:40.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/3963881814,v1:127.0.0.1:6811/3963881814] [v2:127.0.0.1:6812/3963881814,v1:127.0.0.1:6813/3963881814] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:57:40.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:57:40.450 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:57:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:57:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:57:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:57:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:57:40.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:57:40.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:57:40.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:57:40.871 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:57:40.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:57:40.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:40.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:57:40.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-04-01T05:57:40.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-04-01T05:57:40.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-04-01T05:57:40.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:40.976 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:57:41.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-04-01T05:57:41.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-04-01T05:57:41.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-04-01T05:57:41.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:41.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-04-01T05:57:41.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:41.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:57:41.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-04-01T05:57:41.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:41.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-04-01T05:57:41.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-04-01T05:57:41.084 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 450971566085 2024-04-01T05:57:41.084 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:41.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566085 2024-04-01T05:57:41.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:57:42.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:57:42.372 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:42.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-04-01T05:57:42.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:42.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-472446402562 2024-04-01T05:57:42.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:42.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:57:42.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-472446402562 2024-04-01T05:57:42.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:42.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-04-01T05:57:42.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-04-01T05:57:42.670 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 472446402562 2024-04-01T05:57:42.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:57:42.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-04-01T05:57:42.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:57:42.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:42.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:43.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:43.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:57:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:43.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:57:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:57:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:57:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-04-01T05:57:43.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:57:43.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:44.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-04-01T05:57:46.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:57:46.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:57:46.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:57:46.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:57:46.170 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:57:46.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:57:46.172 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:57:46.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:57:46.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:57:46.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:57:46.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:57:46.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:57:46.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:57:46.199 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:46.198+0000 7f63786ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:46.201 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:46.201+0000 7f63786ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:46.202 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:46.202+0000 7f63786ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:46.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:57:46.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:57:46.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:57:46.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:57:46.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:57:46.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:46.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:57:46.496 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:57:46.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:46.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:46.720 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:46.720+0000 7f63786ac640 -1 Falling back to public interface 2024-04-01T05:57:46.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:47.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:47.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:47.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:57:47.776 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:57:47.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:47.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:48.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:49.026 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:49.026+0000 7f63786ac640 -1 osd.1 112 log_to_monitors true 2024-04-01T05:57:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:57:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:57:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:49.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:49.363 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:57:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:57:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:50.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:50.661 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/1179456838,v1:127.0.0.1:6811/1179456838] [v2:127.0.0.1:6812/1179456838,v1:127.0.0.1:6813/1179456838] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:57:50.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:57:50.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:57:50.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:57:50.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:57:50.662 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:57:50.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:57:50.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:57:50.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:57:50.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:57:50.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:57:50.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:57:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:57:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:57:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:57:51.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:51.073 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:57:51.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-04-01T05:57:51.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-04-01T05:57:51.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088' 2024-04-01T05:57:51.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:57:51.178 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:57:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-04-01T05:57:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-04-01T05:57:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2024-04-01T05:57:51.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:51.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566088 2024-04-01T05:57:51.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:51.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:57:51.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566088 2024-04-01T05:57:51.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-04-01T05:57:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-04-01T05:57:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 450971566088 2024-04-01T05:57:51.287 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:57:51.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566088 2024-04-01T05:57:51.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:57:51.590 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-498216206338 2024-04-01T05:57:51.590 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:57:51.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:57:51.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-498216206338 2024-04-01T05:57:51.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:57:51.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-04-01T05:57:51.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-04-01T05:57:51.593 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 498216206338 2024-04-01T05:57:51.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:57:51.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-04-01T05:57:51.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:57:51.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:51.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:52.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:57:52.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:57:52.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:57:52.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:57:52.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:57:52.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:57:52.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:57:52.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:57:52.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:57:52.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:57:52.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:57:52.549 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:57:52.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:57:52.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:57:52.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:57:52.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-04-01T05:57:52.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:57:52.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-04-01T05:57:52.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:57:52.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:57:52.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:57:52.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:57:52.904 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:57:52.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:57:52.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:57:52.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:57:53.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:53.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:57:55.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:57:55.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:57:55.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:57:55.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:57:55.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:57:55.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:57:55.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:57:55.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:57:55.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:55.135+0000 7fb333611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:55.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:55.137+0000 7fb333611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:55.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:55.139+0000 7fb333611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:57:55.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:57:55.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:57:55.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:57:55.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:57:55.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:57:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:57:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:57:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:55.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:55.658 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:55.657+0000 7fb333611640 -1 Falling back to public interface 2024-04-01T05:57:55.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:57:56.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:56.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:56.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:57.957 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:57:57.956+0000 7fb333611640 -1 osd.1 117 log_to_monitors true 2024-04-01T05:57:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:57:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:57:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:58.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:57:59.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:57:59.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:57:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:57:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:57:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:57:59.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:57:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/2547033722,v1:127.0.0.1:6811/2547033722] [v2:127.0.0.1:6812/2547033722,v1:127.0.0.1:6813/2547033722] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:57:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:57:59.675 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:57:59.676 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:57:59.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:57:59.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:57:59.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:57:59.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:57:59.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:57:59.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:58:00.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:58:00.045 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:58:00.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:58:00.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:00.046 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:58:00.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566091 2024-04-01T05:58:00.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566091 2024-04-01T05:58:00.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091' 2024-04-01T05:58:00.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:00.151 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:58:00.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-04-01T05:58:00.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-04-01T05:58:00.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2024-04-01T05:58:00.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:00.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566091 2024-04-01T05:58:00.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:00.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:58:00.257 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566091 2024-04-01T05:58:00.257 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:00.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566091 2024-04-01T05:58:00.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-04-01T05:58:00.259 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 450971566091 2024-04-01T05:58:00.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:00.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566091 2024-04-01T05:58:00.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:58:01.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:58:01.547 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:01.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566091 -lt 450971566091 2024-04-01T05:58:01.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:01.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-519691042818 2024-04-01T05:58:01.845 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:01.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:58:01.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-519691042818 2024-04-01T05:58:01.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-04-01T05:58:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-04-01T05:58:01.849 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 519691042818 2024-04-01T05:58:01.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:02.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-04-01T05:58:02.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:58:02.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:02.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:02.486 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:02.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:58:02.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:02.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:02.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:58:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:58:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:58:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-04-01T05:58:03.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-04-01T05:58:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T05:58:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:58:03.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:58:03.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:03.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:58:05.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:58:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:58:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:58:05.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:58:05.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:58:05.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:58:05.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:58:05.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:58:05.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:05.296+0000 7f01a1981640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:05.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:05.298+0000 7f01a1981640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:05.299 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:05.300+0000 7f01a1981640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:58:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:58:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:58:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:58:05.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:58:05.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:05.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:58:05.564 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:58:05.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:05.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:05.815 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:05.815+0000 7f01a1981640 -1 Falling back to public interface 2024-04-01T05:58:05.873 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:06.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:06.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:06.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:58:06.875 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:58:06.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:06.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:07.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:08.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:08.051+0000 7f01a1981640 -1 osd.1 123 log_to_monitors true 2024-04-01T05:58:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:58:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:58:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:08.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 127 up_thru 121 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/2798700389,v1:127.0.0.1:6811/2798700389] [v2:127.0.0.1:6812/2798700389,v1:127.0.0.1:6813/2798700389] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:58:08.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:58:08.505 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:58:08.505 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:58:08.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:58:08.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:58:08.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:58:08.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:58:08.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:58:08.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:58:08.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:58:08.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:58:08.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:58:08.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:58:08.941 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:58:08.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:58:08.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:08.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:58:09.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-04-01T05:58:09.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-04-01T05:58:09.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094' 2024-04-01T05:58:09.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:09.047 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:58:09.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-04-01T05:58:09.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-04-01T05:58:09.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094 1-545460846594' 2024-04-01T05:58:09.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:09.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566094 2024-04-01T05:58:09.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:09.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:58:09.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566094 2024-04-01T05:58:09.251 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:09.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-04-01T05:58:09.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-04-01T05:58:09.252 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 450971566094 2024-04-01T05:58:09.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:09.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566092 -lt 450971566094 2024-04-01T05:58:09.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:58:10.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:58:10.687 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:10.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566094 2024-04-01T05:58:10.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:10.979 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846594 2024-04-01T05:58:10.980 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:10.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:58:10.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846594 2024-04-01T05:58:10.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:10.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-04-01T05:58:10.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-04-01T05:58:10.982 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 545460846594 2024-04-01T05:58:10.983 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:11.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846594 -lt 545460846594 2024-04-01T05:58:11.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:58:11.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:11.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:11.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:11.625 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:11.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:58:11.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:11.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:11.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:12.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:58:12.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:58:12.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:58:12.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:58:12.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:58:12.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:58:12.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:58:12.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:12.357 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:12.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:12.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:12.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:12.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:12.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:12.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:58:12.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:12.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:12.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:58:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:58:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:58:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:58:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:58:12.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:58:12.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:58:12.987 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:58:12.987 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:58:13.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:58:13.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:58:13.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:58:13.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:58:13.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:58:13.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:58:13.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:13.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:13.569 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:13.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:13.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:13.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:13.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:13.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:58:13.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:13.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:13.849 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:14.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:58:14.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:58:14.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:58:14.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-04-01T05:58:14.206 INFO:tasks.workunit.client.0.smithi067.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-04-01T05:58:14.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:14.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-04-01T05:58:14.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-04-01T05:58:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:58:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:14.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-04-01T05:58:14.208 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-04-01T05:58:14.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:58:14.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:14.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:14.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-04-01T05:58:14.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-04-01T05:58:14.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:58:14.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:14.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:14.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-04-01T05:58:14.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-04-01T05:58:14.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:58:14.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:14.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:14.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-04-01T05:58:14.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-04-01T05:58:14.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:58:14.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:58:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:58:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-04-01T05:58:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:58:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:58:14.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:58:14.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:58:14.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:58:14.521 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:58:16.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:16.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:58:16.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:58:16.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:58:16.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:58:16.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:58:16.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:58:16.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:58:16.312 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:16.311+0000 7f3c6f7e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:16.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:16.313+0000 7f3c6f7e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:16.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:16.315+0000 7f3c6f7e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:16.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:16.838 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:16.838+0000 7f3c6f7e1640 -1 Falling back to public interface 2024-04-01T05:58:16.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:17.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:17.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:58:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:58:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:17.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:18.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:58:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:58:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:19.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:19.318 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:19.318+0000 7f3c6f7e1640 -1 osd.0 129 log_to_monitors true 2024-04-01T05:58:19.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:20.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:20.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:20.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:58:20.524 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:58:20.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:20.524 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:20.895 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/394072184,v1:127.0.0.1:6803/394072184] [v2:127.0.0.1:6804/394072184,v1:127.0.0.1:6805/394072184] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:58:20.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:58:20.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:58:20.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:58:20.896 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:58:20.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:58:20.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:58:20.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:58:20.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:58:20.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:58:20.982 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:58:21.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:58:21.266 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:58:21.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:58:21.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:21.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:58:21.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650370 2024-04-01T05:58:21.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650370 2024-04-01T05:58:21.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370' 2024-04-01T05:58:21.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:21.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:58:21.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846597 2024-04-01T05:58:21.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846597 2024-04-01T05:58:21.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-04-01T05:58:21.477 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:21.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650370 2024-04-01T05:58:21.478 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:21.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:58:21.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650370 2024-04-01T05:58:21.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:21.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650370 2024-04-01T05:58:21.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-04-01T05:58:21.481 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 571230650370 2024-04-01T05:58:21.481 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:21.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650370 -lt 571230650370 2024-04-01T05:58:21.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:21.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846597 2024-04-01T05:58:21.763 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:21.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:58:21.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846597 2024-04-01T05:58:21.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:21.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846597 2024-04-01T05:58:21.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-04-01T05:58:21.765 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 545460846597 2024-04-01T05:58:21.766 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:22.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846596 -lt 545460846597 2024-04-01T05:58:22.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:58:23.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:58:23.050 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:23.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846596 -lt 545460846597 2024-04-01T05:58:23.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:58:24.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:58:24.346 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:24.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846598 -lt 545460846597 2024-04-01T05:58:24.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:58:24.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:24.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:24.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:58:24.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:24.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:25.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:25.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:25.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:25.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:25.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:25.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:58:25.278 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:25.279 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:25.279 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-04-01T05:58:25.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:58:25.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:58:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:58:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-04-01T05:58:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:58:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:58:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:58:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:58:25.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:25.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-04-01T05:58:27.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:58:27.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:58:27.553 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:58:27.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:58:27.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:58:27.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:58:27.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:58:27.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:58:27.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:58:27.582 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:27.582+0000 7fe4ae818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:27.584 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:27.584+0000 7fe4ae818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:27.585 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:27.585+0000 7fe4ae818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:27.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:27.850 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:28.101 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:28.101+0000 7fe4ae818640 -1 Falling back to public interface 2024-04-01T05:58:28.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:29.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:29.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:29.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:58:29.130 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:58:29.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:29.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:29.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:30.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:30.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:30.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:58:30.424 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:58:30.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:30.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:30.550 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:30.550+0000 7fe4ae818640 -1 osd.1 134 log_to_monitors true 2024-04-01T05:58:30.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:31.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:31.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:31.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:58:31.720 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:58:31.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:31.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:32.056 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/2664798317,v1:127.0.0.1:6811/2664798317] [v2:127.0.0.1:6812/2664798317,v1:127.0.0.1:6813/2664798317] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:58:32.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:58:32.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:58:32.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:58:32.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:58:32.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:58:32.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:58:32.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:58:32.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:58:32.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:58:32.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:58:32.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:58:32.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:58:32.440 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:58:32.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:58:32.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:32.440 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:58:32.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650373 2024-04-01T05:58:32.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650373 2024-04-01T05:58:32.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373' 2024-04-01T05:58:32.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:32.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:58:32.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=592705486850 2024-04-01T05:58:32.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 592705486850 2024-04-01T05:58:32.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-04-01T05:58:32.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:32.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650373 2024-04-01T05:58:32.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:32.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:58:32.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650373 2024-04-01T05:58:32.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:32.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650373 2024-04-01T05:58:32.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-04-01T05:58:32.654 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 571230650373 2024-04-01T05:58:32.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:32.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-04-01T05:58:32.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:58:33.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:58:33.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:34.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650373 -lt 571230650373 2024-04-01T05:58:34.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:34.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-592705486850 2024-04-01T05:58:34.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:34.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:58:34.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-592705486850 2024-04-01T05:58:34.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:34.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=592705486850 2024-04-01T05:58:34.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-04-01T05:58:34.230 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 592705486850 2024-04-01T05:58:34.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:34.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 592705486850 -lt 592705486850 2024-04-01T05:58:34.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:58:34.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:34.514 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:34.867 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:34.868 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:35.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:58:35.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:35.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:35.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:35.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:58:35.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:58:35.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:58:35.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-04-01T05:58:35.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:35.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-04-01T05:58:35.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:58:35.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:58:35.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:35.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-04-01T05:58:36.905 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:58:37.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:58:37.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:58:37.424 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:58:37.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:58:37.452 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:37.452+0000 7f7f4ca14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:37.454 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:37.454+0000 7f7f4ca14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:37.455 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:37.455+0000 7f7f4ca14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:37.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:58:37.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:37.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:37.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:37.971+0000 7f7f4ca14640 -1 Falling back to public interface 2024-04-01T05:58:38.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:39.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:39.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:39.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:58:39.002 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:58:39.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:39.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:39.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:40.248 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:40.247+0000 7f7f4ca14640 -1 osd.1 140 log_to_monitors true 2024-04-01T05:58:40.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:40.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:40.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:58:40.297 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:58:40.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:40.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:40.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:41.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:41.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:41.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:58:41.616 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:58:41.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:41.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/2792919327,v1:127.0.0.1:6811/2792919327] [v2:127.0.0.1:6812/2792919327,v1:127.0.0.1:6813/2792919327] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:58:41.899 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:58:41.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:58:41.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:58:41.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:58:41.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:58:41.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:58:41.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:58:41.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:58:41.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:58:41.987 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:58:42.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:58:42.270 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:58:42.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:58:42.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:42.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:58:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650376 2024-04-01T05:58:42.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650376 2024-04-01T05:58:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376' 2024-04-01T05:58:42.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:42.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:58:42.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-04-01T05:58:42.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-04-01T05:58:42.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-04-01T05:58:42.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:42.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650376 2024-04-01T05:58:42.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:42.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:58:42.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650376 2024-04-01T05:58:42.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:42.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650376 2024-04-01T05:58:42.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-04-01T05:58:42.501 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 571230650376 2024-04-01T05:58:42.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:42.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-04-01T05:58:42.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:58:43.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:58:43.785 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:44.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650376 -lt 571230650376 2024-04-01T05:58:44.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:44.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-04-01T05:58:44.079 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:44.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:58:44.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-04-01T05:58:44.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:44.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-04-01T05:58:44.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-04-01T05:58:44.082 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 618475290626 2024-04-01T05:58:44.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:44.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-04-01T05:58:44.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:58:44.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:44.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:44.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:58:44.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:44.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:44.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:44.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:44.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:44.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:44.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:45.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:58:45.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:45.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:45.009 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:45.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:58:45.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:58:45.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:58:45.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:58:45.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-04-01T05:58:45.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:58:45.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:58:45.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:58:45.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:58:45.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-04-01T05:58:46.828 INFO:tasks.workunit.client.0.smithi067.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:58:47.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:58:47.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:58:47.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:58:47.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:58:47.354 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:58:47.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:58:47.354 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:58:47.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:58:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:58:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:58:47.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:58:47.381 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:47.380+0000 7f8e9a01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:47.383 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:47.382+0000 7f8e9a01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:47.385 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:47.384+0000 7f8e9a01b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:47.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:47.900 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:47.899+0000 7f8e9a01b640 -1 Falling back to public interface 2024-04-01T05:58:47.941 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:48.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:48.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:48.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:58:48.943 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:58:48.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:48.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:49.234 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:58:50.178 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:58:50.178+0000 7f8e9a01b640 -1 osd.0 145 log_to_monitors true 2024-04-01T05:58:50.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:58:50.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:58:50.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:58:50.237 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:58:50.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:58:50.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 149 up_thru 144 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/1957969175,v1:127.0.0.1:6803/1957969175] [v2:127.0.0.1:6804/1957969175,v1:127.0.0.1:6805/1957969175] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:58:50.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:58:50.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:58:50.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:58:50.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:58:50.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:58:50.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:58:50.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:58:50.680 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:58:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:58:51.026 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:58:51.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:58:51.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:51.027 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:58:51.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-04-01T05:58:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-04-01T05:58:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-04-01T05:58:51.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:58:51.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:58:51.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-04-01T05:58:51.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-04-01T05:58:51.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-04-01T05:58:51.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:51.275 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-04-01T05:58:51.275 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:51.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:58:51.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-04-01T05:58:51.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-04-01T05:58:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-04-01T05:58:51.278 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 639950127106 2024-04-01T05:58:51.278 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:58:51.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-04-01T05:58:51.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:58:51.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-04-01T05:58:51.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:58:51.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:58:51.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-04-01T05:58:51.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:58:51.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-04-01T05:58:51.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-04-01T05:58:51.596 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 618475290629 2024-04-01T05:58:51.596 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:58:51.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290629 -lt 618475290629 2024-04-01T05:58:51.910 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:58:51.910 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:51.910 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:52.264 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:52.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-01T05:58:52.560 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:52.560 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:52.560 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-04-01T05:58:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-04-01T05:58:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:58:52.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-01T05:58:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:58:52.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:58:53.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:58:53.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:53.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:53.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:53.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:53.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:53.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:53.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:53.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-01T05:58:53.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:53.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:53.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:53.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-04-01T05:58:53.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-01T05:58:53.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:58:53.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:58:53.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:58:53.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:58:53.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:58:53.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:58:53.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:58:54.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:58:54.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:58:54.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:58:54.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:58:54.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:58:54.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:58:54.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:54.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:54.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:54.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:54.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:54.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:54.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:54.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-01T05:58:54.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:54.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:54.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:54.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-04-01T05:58:54.851 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:58:54.852 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:58:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:58:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:58:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-01T05:58:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:58:55.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-01T05:58:55.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:58:55.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:55.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:55.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:55.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:55.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:55.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:55.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:55.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-01T05:58:55.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:55.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:55.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:56.239 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 6 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:58:56.240 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:58:56.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:58:56.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:58:56.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-01T05:58:56.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:58:56.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-01T05:58:57.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:58:57.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:58:57.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:58:57.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:58:57.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:58:57.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:58:57.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:58:57.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:58:57.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:58:57.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:58:57.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:58:57.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:58:58.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:58:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:58:58.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:58:58.077 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:58:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:58:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:58:58.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:58:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-04-01T05:58:59.584 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-04-01T05:58:59.584 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T05:59:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:59:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:59:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:59:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:59:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:59:00.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:59:00.106 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:59:00.107 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:59:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:59:00.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:59:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:59:00.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:59:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:59:00.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:59:00.136 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:00.135+0000 7f054b4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:00.138 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:00.137+0000 7f054b4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:00.139 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:00.139+0000 7f054b4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:59:00.403 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:00.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:00.653 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:00.653+0000 7f054b4d9640 -1 Falling back to public interface 2024-04-01T05:59:00.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:01.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:01.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:01.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:59:01.684 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:59:01.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:01.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:01.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:02.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:02.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:02.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:59:02.976 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:59:02.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:02.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:03.016 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:03.016+0000 7f054b4d9640 -1 osd.1 150 log_to_monitors true 2024-04-01T05:59:03.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:59:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:59:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:04.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:04.687 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/2405820673,v1:127.0.0.1:6811/2405820673] [v2:127.0.0.1:6812/2405820673,v1:127.0.0.1:6813/2405820673] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:59:04.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:59:04.689 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:59:04.689 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:59:04.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:59:04.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:59:04.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:59:04.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:59:04.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:59:04.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:59:04.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:59:04.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:59:04.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:59:05.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:59:05.056 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:59:05.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:59:05.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:05.056 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:59:05.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127109 2024-04-01T05:59:05.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127109 2024-04-01T05:59:05.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109' 2024-04-01T05:59:05.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:05.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:59:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-04-01T05:59:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-04-01T05:59:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-04-01T05:59:05.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:05.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127109 2024-04-01T05:59:05.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:05.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:59:05.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127109 2024-04-01T05:59:05.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:05.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127109 2024-04-01T05:59:05.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-04-01T05:59:05.272 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 639950127109 2024-04-01T05:59:05.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:05.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127109 -lt 639950127109 2024-04-01T05:59:05.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:05.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-04-01T05:59:05.554 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:05.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:59:05.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-04-01T05:59:05.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:05.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-04-01T05:59:05.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-04-01T05:59:05.557 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 661424963586 2024-04-01T05:59:05.557 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:05.840 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-04-01T05:59:05.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:59:05.840 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:05.841 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:06.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:59:06.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:06.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:06.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:06.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:06.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:06.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:06.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:06.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:06.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:06.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:59:06.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:59:06.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:59:06.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:59:06.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-04-01T05:59:06.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-04-01T05:59:06.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T05:59:06.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:59:06.825 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:59:06.826 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:59:06.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:59:06.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:59:06.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:59:07.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:59:07.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:07.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:59:07.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:59:07.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:59:07.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:59:07.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:59:07.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:08.352 INFO:tasks.workunit.client.0.smithi067.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-04-01T05:59:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:59:08.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:59:08.994 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:59:08.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:59:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:59:08.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:59:08.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:59:09.023 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:09.023+0000 7f3bdfe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:09.025 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:09.025+0000 7f3bdfe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:09.027 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:09.027+0000 7f3bdfe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:09.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:09.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:09.547 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:09.547+0000 7f3bdfe0f640 -1 Falling back to public interface 2024-04-01T05:59:09.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:10.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:10.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:10.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:59:10.583 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:59:10.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:10.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:10.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:11.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:11.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:11.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:59:11.881 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:59:11.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:11.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:11.895 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:11.894+0000 7f3bdfe0f640 -1 osd.0 156 log_to_monitors true 2024-04-01T05:59:12.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:13.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:13.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:13.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:59:13.227 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:59:13.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:13.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/2028888228,v1:127.0.0.1:6803/2028888228] [v2:127.0.0.1:6804/2028888228,v1:127.0.0.1:6805/2028888228] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:59:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:59:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:59:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:59:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:59:13.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:59:13.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:59:13.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:59:13.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:59:13.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:59:13.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:59:13.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:59:13.673 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:14.031 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:59:14.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-04-01T05:59:14.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-04-01T05:59:14.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-04-01T05:59:14.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:14.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:59:14.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-04-01T05:59:14.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-04-01T05:59:14.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-04-01T05:59:14.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:14.242 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-04-01T05:59:14.242 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:14.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:59:14.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-04-01T05:59:14.243 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:14.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-04-01T05:59:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-04-01T05:59:14.245 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 687194767362 2024-04-01T05:59:14.245 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:14.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 687194767362 2024-04-01T05:59:14.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:15.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:59:15.531 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:15.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-04-01T05:59:15.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:15.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-04-01T05:59:15.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:15.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:59:15.826 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-04-01T05:59:15.826 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:15.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-04-01T05:59:15.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-04-01T05:59:15.827 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 661424963589 2024-04-01T05:59:15.827 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:16.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-04-01T05:59:16.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:59:16.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:16.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:16.459 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:59:16.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:16.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:16.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:17.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:59:17.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:59:17.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:59:17.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:17.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:59:17.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:59:17.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:59:17.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:59:17.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:17.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-04-01T05:59:18.324 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-04-01T05:59:18.324 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T05:59:18.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:59:18.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:59:18.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:59:18.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:59:18.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:59:18.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:18.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:59:18.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:59:18.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:59:18.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:59:18.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:59:18.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:59:18.912 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:59:18.912 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:59:18.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:59:18.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:59:18.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:59:18.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:59:18.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:59:18.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:59:18.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:59:18.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:59:18.915 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:59:18.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:59:18.916 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:59:18.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:59:18.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:59:18.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:59:18.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:59:18.934 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:18.934+0000 7f6c5f0ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:18.936 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:18.936+0000 7f6c5f0ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:18.938 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:18.938+0000 7f6c5f0ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:19.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:59:19.204 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:59:19.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:19.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:19.456 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:19.456+0000 7f6c5f0ce640 -1 Falling back to public interface 2024-04-01T05:59:19.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:20.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:20.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:20.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:59:20.491 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:59:20.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:20.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:20.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:21.596 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:21.595+0000 7f6c5f0ce640 -1 osd.1 161 log_to_monitors true 2024-04-01T05:59:21.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:21.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:21.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:59:21.785 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:59:21.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:21.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:22.179 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 165 up_thru 160 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/254589357,v1:127.0.0.1:6811/254589357] [v2:127.0.0.1:6812/254589357,v1:127.0.0.1:6813/254589357] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:59:22.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:59:22.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:59:22.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:59:22.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:59:22.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:59:22.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:59:22.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:59:22.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:59:22.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:59:22.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:59:22.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:59:22.269 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:59:22.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:59:22.699 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:59:22.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:59:22.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:22.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:59:22.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767365 2024-04-01T05:59:22.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767365 2024-04-01T05:59:22.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365' 2024-04-01T05:59:22.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:22.841 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:59:22.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603842 2024-04-01T05:59:22.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603842 2024-04-01T05:59:22.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365 1-708669603842' 2024-04-01T05:59:22.948 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:22.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767365 2024-04-01T05:59:22.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:22.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:59:22.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767365 2024-04-01T05:59:22.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:22.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767365 2024-04-01T05:59:22.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2024-04-01T05:59:22.951 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 687194767365 2024-04-01T05:59:22.952 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:23.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767365 2024-04-01T05:59:23.245 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:24.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:59:24.246 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:24.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767365 2024-04-01T05:59:24.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:24.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603842 2024-04-01T05:59:24.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:24.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:59:24.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603842 2024-04-01T05:59:24.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:24.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603842 2024-04-01T05:59:24.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-04-01T05:59:24.576 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 708669603842 2024-04-01T05:59:24.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:24.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603842 -lt 708669603842 2024-04-01T05:59:24.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:59:24.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:24.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:25.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:59:25.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:25.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:25.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:25.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:25.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:25.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:25.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:25.489 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:59:25.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:25.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:25.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:25.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:59:25.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:59:25.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:59:25.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:59:25.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:59:25.841 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:59:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:59:25.940 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:25.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:25.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:25.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:25.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:25.941 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:25.942 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:26.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:59:26.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:26.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:26.222 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:26.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:59:26.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:59:26.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:59:26.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:59:26.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:59:26.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:59:26.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:59:26.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:59:26.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:59:26.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:59:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:59:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:59:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:59:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:27.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:27.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:59:27.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:27.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:27.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:59:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:59:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:59:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T05:59:27.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-04-01T05:59:27.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-04-01T05:59:27.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T05:59:27.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:59:27.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T05:59:27.774 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:59:27.774 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:59:27.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:59:27.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:59:27.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:59:28.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:29.143 INFO:tasks.workunit.client.0.smithi067.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-04-01T05:59:29.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T05:59:29.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:59:29.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:59:29.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:59:29.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:59:29.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:59:29.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:59:29.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T05:59:29.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T05:59:29.747 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T05:59:29.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:59:29.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T05:59:29.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T05:59:29.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:59:29.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:59:29.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:59:29.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:29.773+0000 7f27cf60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:29.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:29.775+0000 7f27cf60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:29.778 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:29.777+0000 7f27cf60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:30.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T05:59:30.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:59:30.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T05:59:30.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:59:30.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:59:30.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:30.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:59:30.041 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:59:30.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:30.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:30.298 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:30.297+0000 7f27cf60b640 -1 Falling back to public interface 2024-04-01T05:59:30.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:31.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:31.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:31.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:59:31.344 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:59:31.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:31.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:31.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:32.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:32.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:32.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:59:32.663 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:59:32.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:32.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:32.710 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:32.709+0000 7f27cf60b640 -1 osd.0 167 log_to_monitors true 2024-04-01T05:59:32.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:33.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:33.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:33.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T05:59:33.958 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T05:59:33.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:33.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T05:59:34.311 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/3664885357,v1:127.0.0.1:6803/3664885357] [v2:127.0.0.1:6804/3664885357,v1:127.0.0.1:6805/3664885357] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T05:59:34.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:59:34.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:59:34.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:59:34.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:59:34.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:59:34.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:59:34.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:59:34.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:59:34.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:59:34.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:59:34.681 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:59:34.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:59:34.682 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:34.682 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:59:34.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407618 2024-04-01T05:59:34.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407618 2024-04-01T05:59:34.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618' 2024-04-01T05:59:34.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:34.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:59:34.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603845 2024-04-01T05:59:34.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603845 2024-04-01T05:59:34.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618 1-708669603845' 2024-04-01T05:59:34.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:34.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407618 2024-04-01T05:59:34.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:34.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:59:34.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407618 2024-04-01T05:59:34.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407618 2024-04-01T05:59:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-04-01T05:59:34.893 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 734439407618 2024-04-01T05:59:34.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-04-01T05:59:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:36.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:59:36.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:36.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407618 -lt 734439407618 2024-04-01T05:59:36.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:36.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603845 2024-04-01T05:59:36.469 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:36.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:59:36.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603845 2024-04-01T05:59:36.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:36.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603845 2024-04-01T05:59:36.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603845' 2024-04-01T05:59:36.472 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 708669603845 2024-04-01T05:59:36.472 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:36.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603845 -lt 708669603845 2024-04-01T05:59:36.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:59:36.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:36.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:37.106 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:37.387 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:59:37.388 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:37.388 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:37.388 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:59:37.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:59:37.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:59:37.742 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:38.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-04-01T05:59:38.854 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-04-01T05:59:38.855 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:59:39.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:59:39.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:59:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:59:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:59:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:59:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:59:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:59:39.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:59:39.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:59:39.422 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:39.422+0000 7ffbbe220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:39.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:39.424+0000 7ffbbe220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:39.426 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:39.426+0000 7ffbbe220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:39.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:59:39.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:39.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:39.953 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:39.953+0000 7ffbbe220640 -1 Falling back to public interface 2024-04-01T05:59:39.971 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:40.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:40.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:40.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:59:40.974 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:59:40.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:40.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:41.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:42.094 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:42.094+0000 7ffbbe220640 -1 osd.1 173 log_to_monitors true 2024-04-01T05:59:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:59:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:59:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:42.691 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 177 up_thru 171 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/3456102491,v1:127.0.0.1:6811/3456102491] [v2:127.0.0.1:6812/3456102491,v1:127.0.0.1:6813/3456102491] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:59:42.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:59:42.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:59:42.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:59:42.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:59:42.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:59:42.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:59:42.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:59:42.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:59:42.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:59:42.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:59:42.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:59:43.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:59:43.325 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:59:43.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:59:43.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:43.326 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:59:43.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407621 2024-04-01T05:59:43.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407621 2024-04-01T05:59:43.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621' 2024-04-01T05:59:43.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:43.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:59:43.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=760209211394 2024-04-01T05:59:43.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 760209211394 2024-04-01T05:59:43.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621 1-760209211394' 2024-04-01T05:59:43.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:43.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407621 2024-04-01T05:59:43.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:43.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:59:43.572 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407621 2024-04-01T05:59:43.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:43.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407621 2024-04-01T05:59:43.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407621' 2024-04-01T05:59:43.574 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 734439407621 2024-04-01T05:59:43.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:43.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407620 -lt 734439407621 2024-04-01T05:59:43.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:44.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:59:44.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:45.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407620 -lt 734439407621 2024-04-01T05:59:45.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:46.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:59:46.155 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:46.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407621 -lt 734439407621 2024-04-01T05:59:46.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:46.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-760209211394 2024-04-01T05:59:46.457 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:46.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:59:46.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-760209211394 2024-04-01T05:59:46.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=760209211394 2024-04-01T05:59:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-04-01T05:59:46.459 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 760209211394 2024-04-01T05:59:46.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:46.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 760209211394 -lt 760209211394 2024-04-01T05:59:46.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:59:46.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:46.744 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:47.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:59:47.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:47.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:47.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:47.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:47.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:47.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:47.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:47.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T05:59:47.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:47.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:47.377 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:47.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T05:59:47.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T05:59:47.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T05:59:47.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T05:59:47.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T05:59:47.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T05:59:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T05:59:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-04-01T05:59:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T05:59:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T05:59:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T05:59:48.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T05:59:48.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:48.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-04-01T05:59:48.885 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-04-01T05:59:48.885 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T05:59:49.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T05:59:49.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T05:59:49.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T05:59:49.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T05:59:49.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T05:59:49.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T05:59:49.416 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T05:59:49.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T05:59:49.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T05:59:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T05:59:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T05:59:49.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T05:59:49.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T05:59:49.444 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:49.444+0000 7f20d7921640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:49.462 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:49.462+0000 7f20d7921640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:49.463 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:49.463+0000 7f20d7921640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:49.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T05:59:49.711 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T05:59:49.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:49.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:49.987 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:49.987+0000 7f20d7921640 -1 Falling back to public interface 2024-04-01T05:59:49.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:50.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:50.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:50.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T05:59:50.994 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T05:59:50.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:50.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:51.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T05:59:52.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T05:59:52.086+0000 7f20d7921640 -1 osd.1 179 log_to_monitors true 2024-04-01T05:59:52.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T05:59:52.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T05:59:52.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T05:59:52.287 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T05:59:52.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T05:59:52.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 183 up_thru 177 down_at 180 last_clean_interval [177,179) [v2:127.0.0.1:6810/3965311628,v1:127.0.0.1:6811/3965311628] [v2:127.0.0.1:6812/3965311628,v1:127.0.0.1:6813/3965311628] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T05:59:52.688 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T05:59:52.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T05:59:52.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T05:59:52.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T05:59:52.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T05:59:52.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T05:59:52.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T05:59:52.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T05:59:52.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T05:59:52.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T05:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T05:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T05:59:53.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T05:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:53.066 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T05:59:53.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407624 2024-04-01T05:59:53.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407624 2024-04-01T05:59:53.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624' 2024-04-01T05:59:53.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T05:59:53.266 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T05:59:53.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015170 2024-04-01T05:59:53.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015170 2024-04-01T05:59:53.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624 1-785979015170' 2024-04-01T05:59:53.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:53.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407624 2024-04-01T05:59:53.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:53.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T05:59:53.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407624 2024-04-01T05:59:53.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:53.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407624 2024-04-01T05:59:53.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2024-04-01T05:59:53.438 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 734439407624 2024-04-01T05:59:53.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T05:59:53.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407624 -lt 734439407624 2024-04-01T05:59:53.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T05:59:53.745 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015170 2024-04-01T05:59:53.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T05:59:53.747 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T05:59:53.747 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015170 2024-04-01T05:59:53.748 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T05:59:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015170 2024-04-01T05:59:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-04-01T05:59:53.748 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 785979015170 2024-04-01T05:59:53.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:54.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 785979015170 2024-04-01T05:59:54.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T05:59:55.074 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:55.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 785979015170 2024-04-01T05:59:55.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T05:59:56.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T05:59:56.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T05:59:56.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015170 -lt 785979015170 2024-04-01T05:59:56.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T05:59:56.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:56.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:57.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T05:59:57.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:57.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:57.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:57.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:57.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:57.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:57.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:57.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:59:57.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:57.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:57.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:57.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:59:57.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T05:59:57.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T05:59:57.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T05:59:57.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:59:57.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T05:59:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:59:57.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:57.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:57.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:57.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:57.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:57.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:57.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:58.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:59:58.031 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:58.031 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:58.031 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:58.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:59:58.383 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:59:58.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:59:58.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:59:58.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T05:59:58.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:59:58.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T05:59:58.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T05:59:58.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T05:59:59.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T05:59:59.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T05:59:59.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T05:59:59.220 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T05:59:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T05:59:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T05:59:59.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T05:59:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T05:59:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T05:59:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T05:59:59.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T05:59:59.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T05:59:59.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T05:59:59.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T05:59:59.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T05:59:59.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-01T05:59:59.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T05:59:59.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-01T06:00:00.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T06:00:00.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:00.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:00.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:00.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:00.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:00.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:00.359 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:00.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:00:00.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:00.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:00.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:00.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-04-01T06:00:00.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-04-01T06:00:00.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-04-01T06:00:00.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-04-01T06:00:00.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-04-01T06:00:00.993 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-04-01T06:00:00.994 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:00:01.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-04-01T06:00:01.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-04-01T06:00:01.374 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0: {} 2024-04-01T06:00:01.375 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-01T06:00:01.392 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1: {} 2024-04-01T06:00:01.392 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:01.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-04-01T06:00:01.403 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:01.700 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:01.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:01.703 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:01.704 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:01.704 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:01.704 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:01.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:01.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:01.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:01.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:01.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-04-01T06:00:01.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:01.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:01.792 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:01.792 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:01.793 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:01.793 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:01.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:01.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:01.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-04-01T06:00:01.884 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:02.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:02.170 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:02.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:02.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:02.259 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:02.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:02.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:02.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:02.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-04-01T06:00:02.351 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:02.664 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-04-01T06:00:02.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:02.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:02.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:02.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:02.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:02.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-04-01T06:00:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-04-01T06:00:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-04-01T06:00:02.847 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-04-01T06:00:02.848 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:03.130 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:03.131 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:03.133 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:03.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:03.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:03.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:03.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:03.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:03.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-04-01T06:00:03.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-04-01T06:00:03.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-04-01T06:00:03.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-04-01T06:00:03.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:03.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:03.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:03.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:03.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:03.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-04-01T06:00:03.858 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0 on osd.1 to scrub 2024-04-01T06:00:03.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:03.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:03.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:56:05.193652+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:04.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:00:05.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:00:05.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:05.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:00:05.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:05.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:05.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:05.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:56:05.193652+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:05.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:00:06.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:00:06.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:06.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:00:06.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:06.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:06.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:06.441 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:56:05.193652+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:00:07.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:00:07.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:07.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:00:07.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:07.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:07.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:07.734 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:08.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:00:04.113388+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-04-01T06:00:08.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.030 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.089 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.098 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.186 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:08.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:08.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-04-01T06:00:08.244 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-04-01T06:00:08.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-04-01T06:00:08.247 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T06:00:08.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-04-01T06:00:08.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-04-01T06:00:08.276 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-04-01T06:00:08.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=183 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T06:00:08.279 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-04-01T06:00:08.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T06:00:08.303 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T06:00:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T06:00:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T06:00:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T06:00:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-04-01T06:00:08.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T06:00:08.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T06:00:08.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T06:00:08.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-04-01T06:00:08.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-04-01T06:00:08.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-04-01T06:00:08.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-04-01T06:00:08.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-04-01T06:00:08.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:00:08.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:00:08.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:00:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:00:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T06:00:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:00:08.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:00:08.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:00:08.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:00:08.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:00:08.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-04-01T06:00:09.787 INFO:tasks.workunit.client.0.smithi067.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-04-01T06:00:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T06:00:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:00:10.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:00:10.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:00:10.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:00:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:00:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:00:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T06:00:10.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:00:10.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:00:10.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:00:10.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:00:10.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:00:10.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:00:10.334 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:10.333+0000 7fabd2011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:10.337 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:10.336+0000 7fabd2011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:10.338 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:10.337+0000 7fabd2011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:10.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:10.853 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:10.852+0000 7fabd2011640 -1 Falling back to public interface 2024-04-01T06:00:10.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:11.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:11.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:11.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:00:11.886 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:00:11.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:11.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:12.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:13.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:13.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:13.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:00:13.183 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:00:13.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:13.183 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:13.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:13.192+0000 7fabd2011640 -1 osd.0 185 log_to_monitors true 2024-04-01T06:00:13.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:14.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:14.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:14.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:00:14.548 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:00:14.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:14.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:14.842 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 189 up_thru 189 down_at 186 last_clean_interval [171,185) [v2:127.0.0.1:6802/1672820273,v1:127.0.0.1:6803/1672820273] [v2:127.0.0.1:6804/1672820273,v1:127.0.0.1:6805/1672820273] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T06:00:14.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:00:14.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:00:14.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:00:14.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:00:14.843 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:00:14.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:00:14.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:00:14.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:00:14.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:00:14.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:00:14.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:00:15.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:00:15.209 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:00:15.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:00:15.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:15.209 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:00:15.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818946 2024-04-01T06:00:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818946 2024-04-01T06:00:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946' 2024-04-01T06:00:15.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:15.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:00:15.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015175 2024-04-01T06:00:15.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015175 2024-04-01T06:00:15.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946 1-785979015175' 2024-04-01T06:00:15.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:15.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818946 2024-04-01T06:00:15.423 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818946 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818946 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 811748818946 2024-04-01T06:00:15.425 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:00:15.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818946 -lt 811748818946 2024-04-01T06:00:15.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:15.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015175 2024-04-01T06:00:15.705 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:15.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:00:15.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015175 2024-04-01T06:00:15.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:15.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015175 2024-04-01T06:00:15.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015175' 2024-04-01T06:00:15.708 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 785979015175 2024-04-01T06:00:15.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:15.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015174 -lt 785979015175 2024-04-01T06:00:15.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:00:16.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:00:16.996 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:17.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015174 -lt 785979015175 2024-04-01T06:00:17.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:00:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T06:00:18.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:18.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015176 -lt 785979015175 2024-04-01T06:00:18.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:00:18.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:18.588 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:00:18.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:18.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:18.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:18.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:18.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:18.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:18.950 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:19.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:00:19.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:19.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:19.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:19.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:00:19.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:00:19.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:00:19.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:00:19.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:00:19.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:00:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:00:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-04-01T06:00:19.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:00:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:00:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:00:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:00:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:00:19.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-04-01T06:00:20.741 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-04-01T06:00:20.741 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T06:00:21.258 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:00:21.259 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:00:21.260 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:00:21.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:00:21.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:00:21.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:00:21.261 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:00:21.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:00:21.288 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:21.288+0000 7fe03e173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:21.290 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:21.290+0000 7fe03e173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:21.291 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:21.291+0000 7fe03e173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:21.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:00:21.554 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:00:21.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:21.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:00:21.806 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:21.806+0000 7fe03e173640 -1 Falling back to public interface 2024-04-01T06:00:21.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:22.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:22.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:22.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:00:22.882 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:00:22.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:22.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:00:23.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:23.884 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:23.884+0000 7fe03e173640 -1 osd.1 191 log_to_monitors true 2024-04-01T06:00:24.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:24.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:24.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:00:24.178 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:00:24.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:24.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:00:24.497 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 195 up_thru 189 down_at 192 last_clean_interval [183,191) [v2:127.0.0.1:6810/3002410110,v1:127.0.0.1:6811/3002410110] [v2:127.0.0.1:6812/3002410110,v1:127.0.0.1:6813/3002410110] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:00:24.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:00:24.499 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:00:24.499 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:00:24.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:00:24.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:00:24.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:00:24.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:00:24.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:00:24.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:00:24.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:00:24.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:00:24.587 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:00:24.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:00:24.992 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:00:24.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:00:24.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:24.992 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:00:25.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818949 2024-04-01T06:00:25.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818949 2024-04-01T06:00:25.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949' 2024-04-01T06:00:25.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:25.133 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:00:25.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622722 2024-04-01T06:00:25.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622722 2024-04-01T06:00:25.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949 1-837518622722' 2024-04-01T06:00:25.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:25.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818949 2024-04-01T06:00:25.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:25.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:00:25.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818949 2024-04-01T06:00:25.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:25.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818949 2024-04-01T06:00:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818949' 2024-04-01T06:00:25.325 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 811748818949 2024-04-01T06:00:25.325 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:00:25.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818949 -lt 811748818949 2024-04-01T06:00:25.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:25.638 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622722 2024-04-01T06:00:25.638 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:25.639 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:00:25.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622722 2024-04-01T06:00:25.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:25.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622722 2024-04-01T06:00:25.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622722' 2024-04-01T06:00:25.641 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 837518622722 2024-04-01T06:00:25.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:25.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622722 -lt 837518622722 2024-04-01T06:00:25.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:00:25.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:25.922 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:26.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:00:26.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:26.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:26.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:26.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:26.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:26.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:26.277 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:26.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T06:00:26.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:26.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:26.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T06:00:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T06:00:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T06:00:26.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T06:00:26.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T06:00:26.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:27.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:27.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T06:00:27.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:27.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:27.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:27.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T06:00:27.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T06:00:28.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T06:00:28.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T06:00:28.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-01T06:00:28.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T06:00:28.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-01T06:00:28.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T06:00:28.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:28.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:28.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:28.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:28.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:28.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:28.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:28.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T06:00:28.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:28.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:28.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-01T06:00:28.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-01T06:00:29.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-01T06:00:29.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-01T06:00:29.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-01T06:00:29.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T06:00:29.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-01T06:00:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T06:00:29.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:29.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:29.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:29.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:29.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:29.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:29.610 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:29.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:00:29.889 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:29.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:29.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:30.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:00:30.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:00:30.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:00:30.244 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-04-01T06:00:30.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:30.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:00:30.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:00:30.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:00:30.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:00:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:00:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:30.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:00:30.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:00:30.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:00:30.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:00:30.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:00:30.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:31.763 INFO:tasks.workunit.client.0.smithi067.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-04-01T06:00:32.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T06:00:32.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:00:32.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:00:32.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:00:32.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:00:32.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:00:32.282 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:00:32.283 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T06:00:32.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:00:32.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:00:32.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:00:32.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:00:32.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:00:32.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:00:32.311 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:32.310+0000 7fd99941b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:32.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:32.313+0000 7fd99941b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:32.315 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:32.315+0000 7fd99941b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:32.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:32.837+0000 7fd99941b640 -1 Falling back to public interface 2024-04-01T06:00:32.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:00:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:00:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:33.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:34.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:35.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:00:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:00:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:35.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:35.188 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:35.188+0000 7fd99941b640 -1 osd.0 197 log_to_monitors true 2024-04-01T06:00:35.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:36.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:36.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:36.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:00:36.616 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:00:36.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:36.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:00:36.911 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 201 up_thru 201 down_at 198 last_clean_interval [189,197) [v2:127.0.0.1:6802/187490135,v1:127.0.0.1:6803/187490135] [v2:127.0.0.1:6804/187490135,v1:127.0.0.1:6805/187490135] exists,up 5e2ee8f3-3eb5-443d-bfb7-8c64ccf04329 2024-04-01T06:00:36.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:00:36.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:00:37.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:00:37.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:00:37.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:00:37.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:00:37.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:00:37.000 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:00:37.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:00:37.285 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:00:37.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:00:37.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:37.285 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:00:37.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426498 2024-04-01T06:00:37.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426498 2024-04-01T06:00:37.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498' 2024-04-01T06:00:37.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:37.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:00:37.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622725 2024-04-01T06:00:37.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622725 2024-04-01T06:00:37.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498 1-837518622725' 2024-04-01T06:00:37.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:37.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426498 2024-04-01T06:00:37.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:37.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:00:37.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426498 2024-04-01T06:00:37.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:37.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426498 2024-04-01T06:00:37.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426498' 2024-04-01T06:00:37.500 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 863288426498 2024-04-01T06:00:37.500 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:00:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426498 -lt 863288426498 2024-04-01T06:00:37.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:37.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622725 2024-04-01T06:00:37.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:37.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:00:37.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622725 2024-04-01T06:00:37.784 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:37.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622725 2024-04-01T06:00:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622725' 2024-04-01T06:00:37.786 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 837518622725 2024-04-01T06:00:37.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:38.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622724 -lt 837518622725 2024-04-01T06:00:38.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:00:39.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:00:39.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:39.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622724 -lt 837518622725 2024-04-01T06:00:39.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:00:40.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T06:00:40.366 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:40.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622726 -lt 837518622725 2024-04-01T06:00:40.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:00:40.660 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:40.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:41.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:41.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:00:41.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:41.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:41.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:00:41.650 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:00:41.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:00:41.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:00:41.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:41.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:00:41.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:00:41.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:00:41.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:00:41.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:00:41.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-04-01T06:00:42.824 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-04-01T06:00:42.824 INFO:tasks.workunit.client.0.smithi067.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-04-01T06:00:43.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T06:00:43.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:00:43.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:00:43.342 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:00:43.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:00:43.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:00:43.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:00:43.344 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T06:00:43.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:00:43.345 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:00:43.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:00:43.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:00:43.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:00:43.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:00:43.370 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:43.370+0000 7fe8e000f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:43.373 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:43.373+0000 7fe8e000f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:43.374 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:43.374+0000 7fe8e000f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:43.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:00:43.887 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:43.887+0000 7fe8e000f640 -1 Falling back to public interface 2024-04-01T06:00:43.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:00:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:00:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:44.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:00:45.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:00:46.002 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:00:46.001+0000 7fe8e000f640 -1 osd.1 203 log_to_monitors true 2024-04-01T06:00:46.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:00:46.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:00:46.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:00:46.207 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:00:46.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:00:46.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:00:46.564 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 207 up_thru 201 down_at 204 last_clean_interval [195,203) [v2:127.0.0.1:6810/1168305059,v1:127.0.0.1:6811/1168305059] [v2:127.0.0.1:6812/1168305059,v1:127.0.0.1:6813/1168305059] exists,up 6283fff3-1f5c-405a-aac0-cd967417ba99 2024-04-01T06:00:46.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:00:46.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:00:46.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:00:46.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:00:46.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:00:46.652 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:00:47.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:00:47.023 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:00:47.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:00:47.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:47.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:00:47.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426501 2024-04-01T06:00:47.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426501 2024-04-01T06:00:47.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501' 2024-04-01T06:00:47.232 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:00:47.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:00:47.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230274 2024-04-01T06:00:47.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230274 2024-04-01T06:00:47.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501 1-889058230274' 2024-04-01T06:00:47.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:47.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426501 2024-04-01T06:00:47.444 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:47.445 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:00:47.445 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426501 2024-04-01T06:00:47.446 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:47.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426501 2024-04-01T06:00:47.447 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426501' 2024-04-01T06:00:47.447 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 863288426501 2024-04-01T06:00:47.447 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:00:47.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426501 -lt 863288426501 2024-04-01T06:00:47.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:00:47.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230274 2024-04-01T06:00:47.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:00:47.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:00:47.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230274 2024-04-01T06:00:47.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:00:47.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230274 2024-04-01T06:00:47.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230274' 2024-04-01T06:00:47.740 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 889058230274 2024-04-01T06:00:47.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:48.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 889058230274 2024-04-01T06:00:48.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:00:49.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:00:49.025 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:49.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 889058230274 2024-04-01T06:00:49.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:00:50.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T06:00:50.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:00:50.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230274 -lt 889058230274 2024-04-01T06:00:50.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:00:50.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:50.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:50.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:51.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-01T06:00:51.246 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:51.246 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:51.246 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:51.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-04-01T06:00:51.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-04-01T06:00:51.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-01T06:00:51.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-04-01T06:00:51.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-01T06:00:51.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-01T06:00:51.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-01T06:00:51.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:00:51.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:00:51.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:00:51.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:00:51.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:00:51.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:00:51.708 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:00:51.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:00:51.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:00:51.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:00:51.985 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:00:52.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-04-01T06:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-04-01T06:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-01T06:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:52.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-04-01T06:00:52.339 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-04-01T06:00:52.340 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:52.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:52.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-04-01T06:00:52.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:52.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-01T06:00:52.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:52.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:52.681 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:52.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:52.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:52.770 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:52.771 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:52.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:52.771 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-04-01T06:00:52.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-04-01T06:00:52.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:52.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:52.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-04-01T06:00:52.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-04-01T06:00:52.861 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:53.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:53.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-04-01T06:00:53.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:53.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-01T06:00:53.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:53.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-01T06:00:53.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:53.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:53.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:53.147 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-01T06:00:53.147 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:53.147 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:53.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:53.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:53.235 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:53.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:53.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-04-01T06:00:53.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:53.236 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:53.236 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:53.236 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:53.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:53.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:53.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-04-01T06:00:53.328 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:53.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:53.614 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:53.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:53.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:53.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:53.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-04-01T06:00:53.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:53.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:53.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:53.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:53.617 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-01T06:00:53.617 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:53.617 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:53.617 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:53.618 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:53.618 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:53.618 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:53.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:53.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:53.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.0.asok 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:53.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-04-01T06:00:53.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-04-01T06:00:53.799 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-04-01T06:00:54.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:54.085 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:54.086 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:54.086 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70255/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-01T06:00:54.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-01T06:00:54.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-osd.1.asok 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-04-01T06:00:54.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70255/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-04-01T06:00:54.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-04-01T06:00:54.389 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0: {} 2024-04-01T06:00:54.390 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0: osd_max_object_size = '' 2024-04-01T06:00:54.407 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1: {} 2024-04-01T06:00:54.407 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1: osd_max_object_size = '' 2024-04-01T06:00:54.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-04-01T06:00:54.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-04-01T06:00:54.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T06:00:54.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:54.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:00:54.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:54.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T06:00:54.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:54.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-04-01T06:00:54.989 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-04-01T06:00:55.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-04-01T05:56:05.193652+0000 last_deep_scrub_stamp 2024-04-01T06:00:55.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T06:00:55.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:55.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:55.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T06:00:55.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:56:05.193652+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:55.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:56.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T06:00:56.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T05:56:05.193652+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:56.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:00:57.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:00:57.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:57.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-04-01T06:00:57.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:57.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:00:57.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:57.575 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-04-01T06:00:57.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:00:55.164976+0000 '>' 2024-04-01T05:56:05.193652+0000 2024-04-01T06:00:57.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:00:57.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-04-01T06:00:57.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.988 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:57.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.005 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.042 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.076 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-04-01T06:00:58.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-04-01T06:00:58.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-04-01T06:00:58.249 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-04-01T06:00:58.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-04-01T06:00:58.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T06:00:58.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-04-01T06:00:58.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-04-01T06:00:58.281 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-04-01T06:00:58.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=207 2024-04-01T06:00:58.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-04-01T06:00:58.284 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-04-01T06:00:58.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T06:00:58.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-04-01T06:00:58.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T06:00:58.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T06:00:58.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T06:00:58.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-04-01T06:00:58.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-04-01T06:00:58.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-04-01T06:00:58.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-04-01T06:00:58.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-04-01T06:00:58.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:58.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:58.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:58.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:58.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T06:00:55.164976+0000 2024-04-01T06:00:58.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-04-01T06:00:58.880 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0 on osd.1 to repair 2024-04-01T06:00:58.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-04-01T06:00:55.164976+0000 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:00:55.164976+0000 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:00:58.893 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:00:59.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:00:55.164976+0000 '>' 2024-04-01T06:00:55.164976+0000 2024-04-01T06:00:59.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:01:00.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:01:00.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:01:00.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:01:00.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:01:00.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:01:00.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:01:00.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:01:00.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:00:55.164976+0000 '>' 2024-04-01T06:00:55.164976+0000 2024-04-01T06:01:00.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:01:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:01:01.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:01:01.465 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:01:01.465 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:01:01.465 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:01:01.465 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:01:01.465 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:01:01.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:00:59.862294+0000 '>' 2024-04-01T06:00:55.164976+0000 2024-04-01T06:01:01.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:01:01.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:01:01.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:01:01.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:01:01.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:01:01.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:01:01.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:01:01.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:01:01.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:01:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:01:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:01:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:01:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:01:02.126 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:01:02.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426505 2024-04-01T06:01:02.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426505 2024-04-01T06:01:02.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505' 2024-04-01T06:01:02.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:01:02.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:01:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230278 2024-04-01T06:01:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230278 2024-04-01T06:01:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505 1-889058230278' 2024-04-01T06:01:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:01:02.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426505 2024-04-01T06:01:02.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:01:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:01:02.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426505 2024-04-01T06:01:02.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:01:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426505 2024-04-01T06:01:02.340 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426505' 2024-04-01T06:01:02.340 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 863288426505 2024-04-01T06:01:02.340 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:01:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426504 -lt 863288426505 2024-04-01T06:01:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:01:03.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:01:03.622 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:01:03.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426505 -lt 863288426505 2024-04-01T06:01:03.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:01:03.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230278 2024-04-01T06:01:03.915 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:01:03.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:01:03.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230278 2024-04-01T06:01:03.917 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:01:03.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230278 2024-04-01T06:01:03.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230278' 2024-04-01T06:01:03.918 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 889058230278 2024-04-01T06:01:03.918 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:01:04.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230278 -lt 889058230278 2024-04-01T06:01:04.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:01:04.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:01:04.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:01:04.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:01:04.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:01:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:01:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:01:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:01:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:01:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:01:04.568 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:01:04.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:01:04.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:01:04.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:01:04.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:01:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:01:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:01:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:01:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-04-01T06:01:05.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-04-01T06:01:05.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-04-01T06:01:05.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-04-01T06:01:05.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-04-01T06:01:05.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-04-01T06:01:05.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-04-01T06:01:05.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-04-01T06:01:05.642 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'csr_pool' removed 2024-04-01T06:01:05.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:01:05.655 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:01:05.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:01:05.762 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:01:05.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:01:05.764 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:01:05.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:01:05.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:01:05.765 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:01:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:01:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:01:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:01:05.767 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:01:05.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:01:05.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:01:05.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:01:05.803 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:01:05.803 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:05.804 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:05.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:01:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:01:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:01:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T06:01:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T06:01:05.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:01:05.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:01:05.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:01:05.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:01:05.808 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:01:05.809 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:01:05.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:01:05.810 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:01:05.810 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:01:05.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:01:05.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:01:05.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:01:05.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:01:05.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:01:05.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:01:05.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:01:05.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:01:05.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:05.815 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:05.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:01:05.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:01:05.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:01:05.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T06:01:05.817 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T06:01:05.817 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:05.817 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:05.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-04-01T06:01:05.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T06:01:05.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-04-01T06:01:06.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T06:01:06.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:01:06.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:01:06.200 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:01:06.201 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:06.201 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:06.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:01:06.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-04-01T06:01:06.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T06:01:06.396 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T06:01:06.396 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:01:06.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:01:06.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T06:01:06.397 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T06:01:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:01:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:01:06.397 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:01:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:01:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:06.398 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:01:06.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:01:06.399 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T06:01:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T06:01:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:01:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:01:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T06:01:06.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:01:06.480 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T06:01:06.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-04-01T06:01:06.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T06:01:06.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T06:01:06.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T06:01:06.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T06:01:06.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T06:01:06.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T06:01:07.370 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:01:07.371 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:01:07.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:01:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-04-01T06:01:07.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:01:07.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:01:07.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=397e932a-3a4f-4900-84b7-c7182a499d84 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 397e932a-3a4f-4900-84b7-c7182a499d84' 2024-04-01T06:01:07.406 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 397e932a-3a4f-4900-84b7-c7182a499d84 2024-04-01T06:01:07.407 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:01:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjTQpmgxAaGRAAl5LQwmVd1wRy6IbmZFk+Mw== 2024-04-01T06:01:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjTQpmgxAaGRAAl5LQwmVd1wRy6IbmZFk+Mw=="}' 2024-04-01T06:01:07.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 397e932a-3a4f-4900-84b7-c7182a499d84 -i td/osd-scrub-repair/0/new.json 2024-04-01T06:01:07.626 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:01:07.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T06:01:07.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjTQpmgxAaGRAAl5LQwmVd1wRy6IbmZFk+Mw== --osd-uuid 397e932a-3a4f-4900-84b7-c7182a499d84 2024-04-01T06:01:07.665 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.665+0000 7f498be14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:07.677 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.677+0000 7f498be14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:07.678 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.678+0000 7f498be14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:07.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.678+0000 7f498be14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:01:07.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.678+0000 7f498be14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:01:07.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.678+0000 7f498be14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:01:07.802 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:07.802+0000 7f498be14640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T06:01:13.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T06:01:13.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:01:13.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T06:01:13.403 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T06:01:13.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:01:13.774 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:01:13.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T06:01:13.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:01:13.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:01:13.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:01:13.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:01:13.798 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:13.798+0000 7fd95c40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:13.800 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:13.800+0000 7fd95c40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:13.801 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:13.801+0000 7fd95c40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:14.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T06:01:14.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:01:14.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:01:14.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:01:14.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:01:14.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:14.061 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:01:14.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:01:14.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:14.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:14.313 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:14.313+0000 7fd95c40d640 -1 Falling back to public interface 2024-04-01T06:01:14.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:15.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:15.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:15.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:01:15.343 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:01:15.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:15.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:15.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:16.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:16.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:16.632 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:01:16.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:01:16.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:16.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:16.638 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:16.634+0000 7fd95c40d640 -1 osd.0 0 log_to_monitors true 2024-04-01T06:01:17.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:18.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:18.067 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:01:18.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:18.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:01:18.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:18.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:18.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:19.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:19.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:19.392 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:01:19.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:01:19.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:19.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:19.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:20.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:01:20.686 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:01:20.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:20.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:20.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:21.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:21.980 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:01:21.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:21.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:01:21.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:21.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:22.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:23.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:01:23.384 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:01:23.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:23.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:23.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:24.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:24.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:24.680 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:01:24.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:01:24.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:24.681 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:24.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:25.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:25.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:25.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:01:25.976 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:01:25.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:25.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:26.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:27.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:27.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:27.358 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:01:27.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:01:27.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:27.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:27.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:28.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:28.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:28.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:01:28.653 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:01:28.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:28.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:29.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:30.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:30.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:30.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:01:30.125 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:01:30.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:30.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:31.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:32.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:32.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:32.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:01:32.015 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:01:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:32.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:32.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:33.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:33.571 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:01:33.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:01:33.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:33.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:34.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:35.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:35.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:35.417 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:01:35.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:01:35.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:35.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:36.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:37.092 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:37.093 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:01:37.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:37.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:01:37.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:37.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:37.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:38.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:38.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:38.869 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:01:38.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:01:38.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:38.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:01:39.168 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3995794186,v1:127.0.0.1:6803/3995794186] [v2:127.0.0.1:6804/3995794186,v1:127.0.0.1:6805/3995794186] exists,up 397e932a-3a4f-4900-84b7-c7182a499d84 2024-04-01T06:01:39.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:01:39.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:01:39.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:01:39.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-04-01T06:01:39.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:01:39.173 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:01:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:01:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e507253-2aac-485b-bc18-4970a28406ee 2024-04-01T06:01:39.176 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 9e507253-2aac-485b-bc18-4970a28406ee 2024-04-01T06:01:39.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9e507253-2aac-485b-bc18-4970a28406ee' 2024-04-01T06:01:39.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:01:39.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDTQpmMPtTCxAA7Ja7YCQ3rcen5Eq6VRQLlA== 2024-04-01T06:01:39.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDTQpmMPtTCxAA7Ja7YCQ3rcen5Eq6VRQLlA=="}' 2024-04-01T06:01:39.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e507253-2aac-485b-bc18-4970a28406ee -i td/osd-scrub-repair/1/new.json 2024-04-01T06:01:39.520 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:01:39.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T06:01:39.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDTQpmMPtTCxAA7Ja7YCQ3rcen5Eq6VRQLlA== --osd-uuid 9e507253-2aac-485b-bc18-4970a28406ee 2024-04-01T06:01:39.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.559+0000 7f01dd1aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:39.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.604+0000 7f01dd1aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:39.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.605+0000 7f01dd1aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:39.606 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.605+0000 7f01dd1aa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:01:39.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.605+0000 7f01dd1aa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:01:39.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.605+0000 7f01dd1aa640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:01:39.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:39.680+0000 7f01dd1aa640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T06:01:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T06:01:45.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:01:45.458 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T06:01:45.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T06:01:45.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:01:45.827 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:01:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T06:01:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:01:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:01:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:01:45.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:01:45.854 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:45.853+0000 7fc9d2624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:45.855 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:45.855+0000 7fc9d2624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:45.856 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:45.856+0000 7fc9d2624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:46.114 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:01:46.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:01:46.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:46.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:46.384 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:46.384+0000 7fc9d2624640 -1 Falling back to public interface 2024-04-01T06:01:46.395 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:47.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:47.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:47.397 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:01:47.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:01:47.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:47.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:47.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:48.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:48.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:48.690 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:01:48.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:01:48.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:48.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:48.837 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:01:48.837+0000 7fc9d2624640 -1 osd.1 0 log_to_monitors true 2024-04-01T06:01:49.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:50.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:50.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:50.039 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:01:50.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:01:50.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:50.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:50.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:51.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:51.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:51.336 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:01:51.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:01:51.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:51.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:51.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:52.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:52.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:52.648 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:01:52.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:01:52.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:52.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:52.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:53.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:53.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:53.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:01:53.994 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:01:53.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:53.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:54.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:55.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:55.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:55.291 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:01:55.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:01:55.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:55.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:55.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:56.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:56.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:56.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:01:56.636 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:01:56.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:56.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:56.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:57.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:57.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:57.975 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:01:57.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:01:57.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:57.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:58.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:01:59.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:01:59.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:01:59.271 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:01:59.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:01:59.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:01:59.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:01:59.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:00.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:00.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:00.619 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:02:00.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:02:00.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:00.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:01.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:02.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:02.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:02.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:02:02.203 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:02:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:02.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:03.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:03.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:03.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:02:04.000 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:02:04.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:04.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:04.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:05.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:05.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:05.861 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:02:05.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:02:05.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:05.862 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:06.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:07.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:07.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:07.238 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:02:07.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:02:07.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:07.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:07.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:08.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:08.722 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:02:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:02:08.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:08.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:09.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:10.341 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:10.341 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:02:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:02:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:10.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:10.663 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:11.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:11.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:11.665 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:02:11.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:02:11.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:11.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:12.008 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/858146418,v1:127.0.0.1:6811/858146418] [v2:127.0.0.1:6812/858146418,v1:127.0.0.1:6813/858146418] exists,up 9e507253-2aac-485b-bc18-4970a28406ee 2024-04-01T06:02:12.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:02:12.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:02:12.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:02:12.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-04-01T06:02:12.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T06:02:12.281 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T06:02:12.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T06:02:12.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T06:02:12.774 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T06:02:12.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:02:13.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T06:02:15.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-04-01T06:02:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:02:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:02:15.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:02:15.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:02:15.578 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:02:15.578 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:02:15.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:02:15.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:02:15.578 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:02:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:02:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:02:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:02:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:02:15.679 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:02:15.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:02:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:02:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:02:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:02:15.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:15.981 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:02:16.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-04-01T06:02:16.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-04-01T06:02:16.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-04-01T06:02:16.093 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:16.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:02:16.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-04-01T06:02:16.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-04-01T06:02:16.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672966' 2024-04-01T06:02:16.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:16.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-04-01T06:02:16.200 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:16.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:02:16.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-04-01T06:02:16.201 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:16.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-04-01T06:02:16.202 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836491 2024-04-01T06:02:16.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-04-01T06:02:16.203 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:16.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-04-01T06:02:16.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:02:17.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:02:17.496 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:17.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-04-01T06:02:17.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:17.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-04-01T06:02:17.791 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:17.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:02:17.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-04-01T06:02:17.792 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:17.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-04-01T06:02:17.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-04-01T06:02:17.794 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672966 2024-04-01T06:02:17.794 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:02:18.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-04-01T06:02:18.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:02:18.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:18.078 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:18.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:02:18.435 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:02:18.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T06:02:18.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:02:18.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:18.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:19.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T06:02:19.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:02:19.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:02:19.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-04-01T06:02:19.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-04-01T06:02:19.495 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'foo' created 2024-04-01T06:02:19.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:02:20.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-04-01T06:02:20.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-04-01T06:02:20.943 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'csr_pool' created 2024-04-01T06:02:20.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:02:21.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-04-01T06:02:21.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:02:21.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:02:21.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:02:21.957 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:02:21.958 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:02:21.958 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:02:21.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:02:21.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:02:21.958 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:02:22.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:02:22.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:02:22.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:02:22.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:02:22.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:02:22.060 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:02:22.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:02:22.471 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:02:22.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:02:22.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:22.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:02:22.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-04-01T06:02:22.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-04-01T06:02:22.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-04-01T06:02:22.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:22.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:02:22.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-04-01T06:02:22.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-04-01T06:02:22.734 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672969' 2024-04-01T06:02:22.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:22.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-04-01T06:02:22.735 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:02:22.736 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-04-01T06:02:22.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-04-01T06:02:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-04-01T06:02:22.738 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836493 2024-04-01T06:02:22.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:23.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-04-01T06:02:23.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:02:24.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:02:24.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:24.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-04-01T06:02:24.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:24.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-04-01T06:02:24.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:24.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:02:24.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-04-01T06:02:24.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:24.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-04-01T06:02:24.333 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672969 2024-04-01T06:02:24.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-01T06:02:24.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:02:24.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-01T06:02:24.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:02:24.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:24.659 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:02:25.053 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:02:25.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:02:25.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:02:25.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:25.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:25.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:02:25.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:02:25.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:02:25.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T06:02:25.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T06:02:26.091 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T06:02:26.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T06:02:26.619 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:02:26.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:02:26.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:02:26.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:02:26.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-04-01T06:02:26.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T06:02:27.075 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T06:02:27.456 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T06:02:27.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T06:02:27.976 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:02:27.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:02:27.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:02:27.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:02:28.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-04-01T06:02:28.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-04-01T06:02:28.465 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-04-01T06:02:28.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-04-01T06:02:28.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-04-01T06:02:28.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-04-01T06:02:28.466 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:02:28.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-04-01T06:02:28.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-04-01T06:02:28.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-04-01T06:02:28.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-04-01T06:02:28.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-04-01T06:02:28.753 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T06:02:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-04-01T06:02:29.039 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-04-01T06:02:29.249 INFO:tasks.workunit.client.0.smithi067.stdout:created pool csr_pool snap snap1 2024-04-01T06:02:29.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-04-01T06:02:29.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-04-01T06:02:29.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-04-01T06:02:29.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-04-01T06:02:29.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-04-01T06:02:29.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-04-01T06:02:29.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-04-01T06:02:29.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-04-01T06:02:29.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:02:29.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:02:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:02:29.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:02:29.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:02:29.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:02:31.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:31.812 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:02:31.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:02:31.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:02:31.814 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T06:02:31.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:02:31.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:02:31.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:02:31.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:02:31.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:02:31.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:02:31.841 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:31.841+0000 7f2ab53ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:31.843 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:31.843+0000 7f2ab53ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:31.844 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:31.844+0000 7f2ab53ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:32.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:32.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:32.361+0000 7f2ab53ac640 -1 Falling back to public interface 2024-04-01T06:02:32.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:02:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:02:33.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:33.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:33.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:34.727 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:34.775 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:34.775+0000 7f2ab53ac640 -1 osd.1 31 log_to_monitors true 2024-04-01T06:02:35.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:36.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:36.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:36.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:02:36.172 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:02:36.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:36.172 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1740902670,v1:127.0.0.1:6811/1740902670] [v2:127.0.0.1:6812/1740902670,v1:127.0.0.1:6813/1740902670] exists,up 9e507253-2aac-485b-bc18-4970a28406ee 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:02:36.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:02:36.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:02:36.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:02:36.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:02:36.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:02:36.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:02:36.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:02:36.586 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:02:36.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:02:36.892 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:02:36.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:02:36.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:36.892 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:02:36.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-04-01T06:02:36.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-04-01T06:02:36.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-04-01T06:02:36.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:36.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:02:37.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-04-01T06:02:37.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-04-01T06:02:37.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-150323855362' 2024-04-01T06:02:37.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:37.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-04-01T06:02:37.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:37.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:02:37.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-04-01T06:02:37.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:37.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-04-01T06:02:37.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-04-01T06:02:37.107 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 21474836497 2024-04-01T06:02:37.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:37.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-04-01T06:02:37.388 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:02:38.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:02:38.390 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:38.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-04-01T06:02:38.685 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:38.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-04-01T06:02:38.685 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:38.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:02:38.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-04-01T06:02:38.688 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:38.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-04-01T06:02:38.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-04-01T06:02:38.688 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 150323855362 2024-04-01T06:02:38.689 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:02:38.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-04-01T06:02:38.972 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:02:38.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:38.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:39.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:02:39.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:02:39.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:02:39.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:02:39.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:02:39.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:02:39.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:02:39.328 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:02:39.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:02:39.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:02:39.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:39.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:39.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:02:39.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:02:39.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:02:39.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-04-01T06:02:39.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-04-01T06:02:39.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-04-01T06:02:39.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-04-01T06:02:39.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-04-01T06:02:39.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-04-01T06:02:40.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-04-01T06:02:40.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:02:40.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:02:40.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:02:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:02:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-04-01T06:02:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:02:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:02:40.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:02:40.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:02:40.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:02:40.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-04-01T06:02:42.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:02:42.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:02:42.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:02:42.321 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T06:02:42.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:02:42.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:02:42.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:02:42.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:02:42.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:02:42.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:02:42.348 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:42.348+0000 7f34ca48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:42.351+0000 7f34ca48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:42.352 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:42.352+0000 7f34ca48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:42.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:02:42.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:42.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:02:42.866 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:42.866+0000 7f34ca48b640 -1 Falling back to public interface 2024-04-01T06:02:42.917 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:43.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:43.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:43.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:02:43.919 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:02:43.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:43.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:02:44.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:45.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:45.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:45.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:02:45.214 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:02:45.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:45.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:02:45.229 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:45.229+0000 7f34ca48b640 -1 osd.0 37 log_to_monitors true 2024-04-01T06:02:45.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:02:46.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:02:46.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:02:46.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:02:46.561 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:02:46.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:02:46.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:02:46.854 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/2008764420,v1:127.0.0.1:6803/2008764420] [v2:127.0.0.1:6804/2008764420,v1:127.0.0.1:6805/2008764420] exists,up 397e932a-3a4f-4900-84b7-c7182a499d84 2024-04-01T06:02:46.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:02:46.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:02:46.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:02:46.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:02:46.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:02:46.856 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:02:46.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:02:46.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:02:46.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:02:46.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:02:46.945 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:02:46.945 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:02:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:02:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:02:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:02:47.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:47.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:02:47.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-04-01T06:02:47.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-04-01T06:02:47.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659138' 2024-04-01T06:02:47.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:02:47.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:02:47.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-04-01T06:02:47.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-04-01T06:02:47.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659138 1-150323855365' 2024-04-01T06:02:47.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:47.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659138 2024-04-01T06:02:47.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:47.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:02:47.437 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659138 2024-04-01T06:02:47.438 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:47.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-04-01T06:02:47.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659138' 2024-04-01T06:02:47.439 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 176093659138 2024-04-01T06:02:47.439 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:02:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-04-01T06:02:47.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:02:47.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-04-01T06:02:47.726 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:02:47.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:02:47.727 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-04-01T06:02:47.727 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:02:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-04-01T06:02:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-04-01T06:02:47.728 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 150323855365 2024-04-01T06:02:47.729 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:02:48.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-04-01T06:02:48.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:02:48.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:48.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:02:48.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:02:48.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-01T06:02:48.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:02:48.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:02:48.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:02:49.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-04-01T06:02:49.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:02:49.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:02:49.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:02:49.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:02:49.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-04-01T06:02:20.845748+0000 2024-04-01T06:02:49.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-04-01T06:02:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 3.0 on osd.1 to scrub 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-04-01T06:02:20.845748+0000 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:02:20.845748+0000 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:02:49.570 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:02:49.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:02:20.845748+0000 '>' 2024-04-01T06:02:20.845748+0000 2024-04-01T06:02:49.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:02:50.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:02:50.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:02:50.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:02:50.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:02:50.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:02:50.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:02:50.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:02:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:02:20.845748+0000 '>' 2024-04-01T06:02:20.845748+0000 2024-04-01T06:02:51.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:02:52.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:02:52.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:02:52.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-04-01T06:02:52.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-04-01T06:02:52.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:02:52.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:02:52.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-04-01T06:02:52.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:02:49.578708+0000 '>' 2024-04-01T06:02:20.845748+0000 2024-04-01T06:02:52.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:02:52.466 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-04-01T06:02:52.495 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-04-01T06:02:52.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-04-01T06:02:52.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-04-01T06:02:52.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-04-01T06:02:52.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T06:02:52.564 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T06:02:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T06:02:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T06:02:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-04-01T06:02:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-04-01T06:02:52.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T06:02:52.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: . as $in 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: | if type == "object" then 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: reduce keys[] as $key 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: elif type == "array" then map( walk(f) ) | f 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: else f 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr: end; 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.version) else . end) 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-04-01T06:02:52.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-04-01T06:02:52.611 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-04-01T06:02:52.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-04-01T06:02:52.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-04-01T06:02:52.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-04-01T06:02:52.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-04-01T06:02:52.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-04-01T06:02:52.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-04-01T06:02:52.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-04-01T06:02:52.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-04-01T06:02:52.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-04-01T06:02:52.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-04-01T06:02:52.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-04-01T06:02:52.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-04-01T06:02:52.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-04-01T06:02:52.998 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'csr_pool' removed 2024-04-01T06:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T06:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:02:53.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:02:53.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:02:53.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:02:53.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:02:53.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:02:53.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:02:53.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:02:53.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:02:53.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:02:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:02:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:02:53.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:02:53.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:02:53.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:02:53.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:02:53.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:02:53.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:02:53.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:53.136 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:53.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:02:53.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:02:53.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:02:53.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:02:53.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:02:53.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:02:53.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:02:53.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:02:53.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:02:53.142 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:02:53.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:02:53.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:02:53.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:02:53.144 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:02:53.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:02:53.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:02:53.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:02:53.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:02:53.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:53.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:53.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:02:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:02:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:02:53.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T06:02:53.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T06:02:53.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:53.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:53.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-04-01T06:02:53.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T06:02:53.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T06:02:53.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T06:02:53.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T06:02:53.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T06:02:53.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T06:02:53.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T06:02:53.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:02:53.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:02:53.505 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:02:53.505 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:53.506 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:53.506 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:53.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T06:02:53.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T06:02:53.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T06:02:53.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:02:53.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:02:53.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T06:02:53.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:02:53.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:02:53.776 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:02:53.777 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:02:53.778 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:02:53.778 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:53.778 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:53.778 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:02:53.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:02:53.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T06:02:53.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-04-01T06:02:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T06:02:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T06:02:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T06:02:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T06:02:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T06:02:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:54.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:02:54.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:02:54.696 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-04-01T06:02:54.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:02:54.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-04-01T06:02:54.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:02:54.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:02:54.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:02:54.701 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:02:54.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:02:54.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:02:54.703 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:02:54.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a3a3720-bb5c-46d3-8170-c36f9349ff99 2024-04-01T06:02:54.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8a3a3720-bb5c-46d3-8170-c36f9349ff99' 2024-04-01T06:02:54.704 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 8a3a3720-bb5c-46d3-8170-c36f9349ff99 2024-04-01T06:02:54.704 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:02:54.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAOTgpmtnzXKhAAR6p+RGFHO1yIZQLAIrLdzQ== 2024-04-01T06:02:54.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAOTgpmtnzXKhAAR6p+RGFHO1yIZQLAIrLdzQ=="}' 2024-04-01T06:02:54.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a3a3720-bb5c-46d3-8170-c36f9349ff99 -i td/osd-scrub-repair/0/new.json 2024-04-01T06:02:54.922 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:02:54.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T06:02:54.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOTgpmtnzXKhAAR6p+RGFHO1yIZQLAIrLdzQ== --osd-uuid 8a3a3720-bb5c-46d3-8170-c36f9349ff99 2024-04-01T06:02:54.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:54.961+0000 7ff562123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:54.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:54.973+0000 7ff562123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:54.974 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:54.974+0000 7ff562123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:02:54.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:54.975+0000 7ff562123640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:02:54.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:54.975+0000 7ff562123640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:02:54.975 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:54.975+0000 7ff562123640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:02:55.065 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:02:55.065+0000 7ff562123640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T06:03:00.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T06:03:00.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:03:00.676 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T06:03:00.676 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T06:03:00.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:03:01.058 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:03:01.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T06:03:01.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:03:01.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:03:01.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:03:01.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:03:01.086 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:01.085+0000 7f475a938640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:01.088 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:01.088+0000 7f475a938640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:01.089 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:01.089+0000 7f475a938640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:01.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T06:03:01.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:03:01.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:03:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:03:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:03:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:03:01.345 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:03:01.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:01.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:01.604 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:01.604+0000 7f475a938640 -1 Falling back to public interface 2024-04-01T06:03:01.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:02.626 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:03:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:03:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:02.916 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:03.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:03.918 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:03.918 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:03:03.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:03:03.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:03.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:03.939 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:03.938+0000 7f475a938640 -1 osd.0 0 log_to_monitors true 2024-04-01T06:03:04.381 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:05.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:05.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:05.383 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:03:05.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:03:05.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:05.384 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:05.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:03:06.730 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:03:06.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:06.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:07.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:08.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:08.026 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:03:08.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:03:08.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:08.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:08.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:09.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:09.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:09.326 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:03:09.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:03:09.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:09.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:09.687 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:10.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:10.688 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:10.688 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:03:10.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:03:10.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:10.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:10.981 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:11.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:11.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:11.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:03:11.983 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:03:11.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:11.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:12.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:13.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:13.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:13.281 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:03:13.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:03:13.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:13.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:13.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:14.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:14.672 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:14.672 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:03:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:03:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:14.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:14.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:15.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:15.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:15.966 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:03:15.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:03:15.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:15.967 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:16.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:17.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:17.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:17.408 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:03:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:03:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:17.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:18.130 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:19.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:19.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:19.132 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:03:19.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:03:19.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:19.133 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:19.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:20.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:20.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:20.433 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:03:20.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:03:20.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:20.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:20.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:21.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:21.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:21.877 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:03:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:03:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:21.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:22.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:23.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:23.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:23.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:03:23.556 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:03:23.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:23.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:24.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:25.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:25.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:25.299 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:03:25.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:03:25.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:25.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:26.010 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:27.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:27.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:27.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:03:27.012 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:03:27.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:27.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:03:27.329 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1868265269,v1:127.0.0.1:6803/1868265269] [v2:127.0.0.1:6804/1868265269,v1:127.0.0.1:6805/1868265269] exists,up 8a3a3720-bb5c-46d3-8170-c36f9349ff99 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:03:27.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:03:27.334 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:03:27.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:03:27.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:03:27.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:03:27.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:03:27.336 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:03:27.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ea6b8c0-0c98-4eaf-828a-f43f122c336a 2024-04-01T06:03:27.336 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7ea6b8c0-0c98-4eaf-828a-f43f122c336a' 2024-04-01T06:03:27.337 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 7ea6b8c0-0c98-4eaf-828a-f43f122c336a 2024-04-01T06:03:27.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:03:27.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvTgpmblfrFBAAfnCjrgJuN/ZZ6aRFQukQKw== 2024-04-01T06:03:27.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvTgpmblfrFBAAfnCjrgJuN/ZZ6aRFQukQKw=="}' 2024-04-01T06:03:27.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ea6b8c0-0c98-4eaf-828a-f43f122c336a -i td/osd-scrub-repair/1/new.json 2024-04-01T06:03:27.712 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:03:27.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T06:03:27.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvTgpmblfrFBAAfnCjrgJuN/ZZ6aRFQukQKw== --osd-uuid 7ea6b8c0-0c98-4eaf-828a-f43f122c336a 2024-04-01T06:03:27.752 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.752+0000 7fa2979bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:27.789 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.789+0000 7fa2979bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.790+0000 7fa2979bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.790+0000 7fa2979bb640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:03:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.790+0000 7fa2979bb640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:03:27.790 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.790+0000 7fa2979bb640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:03:27.883 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:27.882+0000 7fa2979bb640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T06:03:33.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T06:03:33.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:03:33.894 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T06:03:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T06:03:33.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:03:34.267 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:03:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T06:03:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:03:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:03:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:03:34.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:03:34.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:34.293+0000 7f576b00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:34.295 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:34.295+0000 7f576b00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:34.296 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:34.296+0000 7f576b00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:03:34.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:03:34.561 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:03:34.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:34.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:34.818 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:34.818+0000 7f576b00f640 -1 Falling back to public interface 2024-04-01T06:03:34.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:35.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:35.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:35.899 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:03:35.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:03:35.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:35.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:36.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:37.189 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:03:37.189+0000 7f576b00f640 -1 osd.1 0 log_to_monitors true 2024-04-01T06:03:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:37.195 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:03:37.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:03:37.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:37.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:37.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:38.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:38.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:38.506 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:03:38.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:03:38.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:38.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:38.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:39.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:39.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:39.802 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:03:39.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:03:39.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:39.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:40.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:41.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:41.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:41.163 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:03:41.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:03:41.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:41.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:41.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:42.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:42.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:42.459 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:03:42.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:03:42.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:42.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:42.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:43.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:43.802 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:03:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:03:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:43.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:44.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:45.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:45.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:45.196 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:03:45.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:03:45.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:45.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:45.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:46.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:46.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:46.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:03:46.492 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:03:46.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:46.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:46.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:47.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:47.794 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:47.794 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:03:47.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:03:47.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:47.795 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:48.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:49.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:49.198 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:49.198 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:03:49.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:03:49.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:49.199 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:49.706 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:50.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:50.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:50.708 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:03:50.708 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:03:50.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:50.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:51.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:52.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:52.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:52.749 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:03:52.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:03:52.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:52.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:53.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:54.078 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:54.078 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:03:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:03:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:54.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:55.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:55.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:55.510 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:03:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:03:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:55.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:56.262 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:57.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:57.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:57.264 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:03:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:03:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:57.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:03:58.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:03:58.882 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:03:58.882 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:03:58.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:03:58.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:03:58.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:03:59.236 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:00.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:00.238 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:04:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:04:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:00.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:04:00.528 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2632506258,v1:127.0.0.1:6811/2632506258] [v2:127.0.0.1:6812/2632506258,v1:127.0.0.1:6813/2632506258] exists,up 7ea6b8c0-0c98-4eaf-828a-f43f122c336a 2024-04-01T06:04:00.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:04:00.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:04:00.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:04:00.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:04:00.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:04:00.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:04:00.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T06:04:00.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:04:00.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94c73cd5-ccc6-4d3b-b476-f670fdf32d41 2024-04-01T06:04:00.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 94c73cd5-ccc6-4d3b-b476-f670fdf32d41' 2024-04-01T06:04:00.536 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 94c73cd5-ccc6-4d3b-b476-f670fdf32d41 2024-04-01T06:04:00.536 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:04:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQTgpmkCDMIBAAE0imdh2ZM2uNKsyADtp1nw== 2024-04-01T06:04:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQTgpmkCDMIBAAE0imdh2ZM2uNKsyADtp1nw=="}' 2024-04-01T06:04:00.550 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94c73cd5-ccc6-4d3b-b476-f670fdf32d41 -i td/osd-scrub-repair/2/new.json 2024-04-01T06:04:00.972 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:04:00.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T06:04:00.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQTgpmkCDMIBAAE0imdh2ZM2uNKsyADtp1nw== --osd-uuid 94c73cd5-ccc6-4d3b-b476-f670fdf32d41 2024-04-01T06:04:01.012 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.012+0000 7f538ea37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:01.056 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.056+0000 7f538ea37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:01.057 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.057+0000 7f538ea37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:01.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.057+0000 7f538ea37640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:04:01.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.057+0000 7f538ea37640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:04:01.058 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.057+0000 7f538ea37640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:04:01.131 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:01.131+0000 7f538ea37640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T06:04:07.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T06:04:07.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:04:07.196 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T06:04:07.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T06:04:07.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:04:07.565 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T06:04:07.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T06:04:07.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:04:07.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:04:07.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:04:07.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:04:07.591 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:07.591+0000 7faae1b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:07.593 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:07.593+0000 7faae1b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:07.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:07.594+0000 7faae1b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:07.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T06:04:07.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:04:07.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T06:04:07.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:04:07.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:04:07.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:07.869 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:04:07.870 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:04:07.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:07.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:08.118 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:08.118+0000 7faae1b2a640 -1 Falling back to public interface 2024-04-01T06:04:08.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:09.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:09.155 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:09.155 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:04:09.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:04:09.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:09.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:09.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:10.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:10.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:10.451 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:04:10.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:04:10.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:10.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:10.490 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:10.490+0000 7faae1b2a640 -1 osd.2 0 log_to_monitors true 2024-04-01T06:04:10.798 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:11.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:11.799 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:11.800 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:04:11.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:04:11.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:11.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:12.095 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:13.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:13.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:13.096 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:04:13.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:04:13.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:13.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:13.423 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:04:14.425 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:04:14.425 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:14.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:14.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:15.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:15.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:15.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:04:15.720 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:04:15.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:15.720 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:16.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:17.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:17.043 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:04:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:04:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:17.044 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:17.467 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:18.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:18.468 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:18.469 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:04:18.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:04:18.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:18.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:18.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:19.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:19.763 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:19.764 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:04:19.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:04:19.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:19.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:20.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:21.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:21.060 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:04:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:04:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:21.061 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:21.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:22.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:22.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:22.452 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:04:22.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:04:22.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:22.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:22.990 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:23.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:23.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:23.992 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:04:23.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:04:23.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:23.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:24.513 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:25.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:25.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:25.515 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:04:25.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:04:25.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:25.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:26.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:27.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:27.177 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:27.177 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:04:27.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:04:27.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:27.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:27.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:04:28.832 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:04:28.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:28.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:29.538 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:30.539 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:30.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:30.540 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:04:30.540 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:04:30.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:30.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:31.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:32.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:32.113 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:04:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:04:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:32.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:32.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:33.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:33.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:33.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:04:33.601 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:04:33.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:33.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:04:33.907 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2428676372,v1:127.0.0.1:6819/2428676372] [v2:127.0.0.1:6820/2428676372,v1:127.0.0.1:6821/2428676372] exists,up 94c73cd5-ccc6-4d3b-b476-f670fdf32d41 2024-04-01T06:04:33.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:04:33.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:04:33.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:04:33.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-04-01T06:04:33.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T06:04:34.182 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T06:04:34.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T06:04:34.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T06:04:34.580 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T06:04:34.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:04:35.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T06:04:37.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-04-01T06:04:37.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:04:37.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:04:37.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:04:37.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:04:37.891 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:04:37.891 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:04:37.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:04:37.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:04:37.891 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:04:37.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:04:37.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:04:37.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:04:37.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:04:37.992 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:04:37.992 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:04:38.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:04:38.310 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:04:38.310 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:04:38.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:04:38.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:04:38.311 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:04:38.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-04-01T06:04:38.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-04-01T06:04:38.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-04-01T06:04:38.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:04:38.416 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:04:38.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-04-01T06:04:38.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-04-01T06:04:38.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672970' 2024-04-01T06:04:38.522 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:04:38.522 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:04:38.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-04-01T06:04:38.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-04-01T06:04:38.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672970 2-64424509446' 2024-04-01T06:04:38.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:04:38.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-04-01T06:04:38.628 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:04:38.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:04:38.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-04-01T06:04:38.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:04:38.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-04-01T06:04:38.630 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836498 2024-04-01T06:04:38.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-04-01T06:04:38.631 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:04:38.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-04-01T06:04:38.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:04:39.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:04:39.914 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:04:40.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-04-01T06:04:40.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:04:40.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-04-01T06:04:40.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:04:40.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:04:40.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-04-01T06:04:40.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:04:40.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-04-01T06:04:40.213 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672970 2024-04-01T06:04:40.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-04-01T06:04:40.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:04:40.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-04-01T06:04:40.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:04:40.495 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-04-01T06:04:40.495 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:04:40.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:04:40.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-04-01T06:04:40.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:04:40.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-04-01T06:04:40.498 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509446 2024-04-01T06:04:40.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-01T06:04:40.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:04:40.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-04-01T06:04:40.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:04:40.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:04:40.782 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:04:41.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T06:04:41.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:04:41.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:04:41.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:04:41.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:04:41.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:04:41.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:04:41.137 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:04:41.413 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T06:04:41.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:04:41.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:04:41.414 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T06:04:41.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-01T06:04:42.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T06:04:42.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T06:04:42.919 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T06:04:42.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:04:43.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-04-01T06:04:43.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T06:04:43.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:04:43.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:04:43.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:04:43.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:04:43.934 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:04:43.934 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:04:43.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:04:43.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:04:43.934 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:04:44.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:04:44.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:04:44.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:04:44.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:04:44.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:04:44.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:04:44.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:04:44.320 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:04:44.320 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:04:44.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:04:44.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:04:44.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:04:44.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-04-01T06:04:44.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-04-01T06:04:44.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-04-01T06:04:44.426 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:04:44.426 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:04:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-04-01T06:04:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-04-01T06:04:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672972' 2024-04-01T06:04:44.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:04:44.612 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:04:44.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-04-01T06:04:44.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-04-01T06:04:44.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672972 2-64424509449' 2024-04-01T06:04:44.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:04:44.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-04-01T06:04:44.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:04:44.719 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:04:44.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-04-01T06:04:44.720 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:04:44.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-04-01T06:04:44.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-01T06:04:44.721 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836500 2024-04-01T06:04:44.722 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:04:45.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-04-01T06:04:45.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:04:45.066 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-04-01T06:04:45.066 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:04:45.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:04:45.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-04-01T06:04:45.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:04:45.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-04-01T06:04:45.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-04-01T06:04:45.070 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672972 2024-04-01T06:04:45.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:04:45.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672972 2024-04-01T06:04:45.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:04:45.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-04-01T06:04:45.355 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:04:45.356 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:04:45.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-04-01T06:04:45.356 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:04:45.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-04-01T06:04:45.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-01T06:04:45.358 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509449 2024-04-01T06:04:45.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:04:45.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-01T06:04:45.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:04:45.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:04:45.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:04:45.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T06:04:45.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:04:45.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:04:45.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:04:45.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:04:45.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:04:45.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:04:45.995 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:04:46.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T06:04:46.278 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:04:46.278 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:04:46.278 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:04:46.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T06:04:46.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T06:04:47.025 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T06:04:47.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T06:04:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:04:47.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:04:47.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:04:47.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:04:47.824 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-04-01T06:04:47.825 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T06:04:47.825 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-04-01T06:04:47.825 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-04-01T06:04:47.826 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:04:48.112 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-01T06:04:48.112 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:04:48.112 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T06:04:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T06:04:48.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T06:04:48.499 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T06:04:48.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T06:04:48.985 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:04:48.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:04:48.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:04:48.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:04:49.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-04-01T06:04:49.298 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T06:04:49.299 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-04-01T06:04:49.299 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-04-01T06:04:49.299 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:04:49.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-01T06:04:49.604 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:04:49.604 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:04:49.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-01T06:04:49.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-04-01T06:04:49.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-04-01T06:04:49.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-04-01T06:04:49.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:04:49.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:04:49.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:04:49.606 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:04:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:04:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:04:49.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:04:49.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:04:49.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:04:49.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:04:49.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:04:49.913 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:04:49.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:04:49.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:04:49.913 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:04:51.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:04:51.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:04:51.222 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T06:04:51.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:04:51.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:04:51.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:04:51.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:04:51.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:04:51.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:04:51.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-04-01T06:04:51.531 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-04-01T06:04:52.630 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-04-01T06:04:53.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-04-01T06:04:53.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-04-01T06:04:54.237 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-04-01T06:04:54.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-04-01T06:04:54.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-04-01T06:04:55.454 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-04-01T06:04:55.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-04-01T06:04:55.969 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-04-01T06:04:57.109 INFO:tasks.workunit.client.0.smithi067.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-04-01T06:04:57.624 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:04:57.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:04:57.629 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:04:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T06:04:57.631 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:04:57.632 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:04:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:04:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:04:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:04:57.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:04:57.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:57.657+0000 7fa1a72d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:57.660 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:57.660+0000 7fa1a72d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:57.661 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:57.661+0000 7fa1a72d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:57.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:04:58.179 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:04:58.179+0000 7fa1a72d9640 -1 Falling back to public interface 2024-04-01T06:04:58.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:04:59.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:04:59.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:04:59.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:04:59.217 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:04:59.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:04:59.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:04:59.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:00.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:00.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:00.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:05:00.510 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:05:00.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:00.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:00.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:00.565+0000 7fa1a72d9640 -1 osd.0 34 log_to_monitors true 2024-04-01T06:05:00.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:01.820 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:05:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:05:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:01.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/2700236178,v1:127.0.0.1:6803/2700236178] [v2:127.0.0.1:6804/2700236178,v1:127.0.0.1:6805/2700236178] exists,up 8a3a3720-bb5c-46d3-8170-c36f9349ff99 2024-04-01T06:05:02.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:05:02.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:05:02.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:05:02.120 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:05:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:05:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:05:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:05:02.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:05:02.142 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:02.142+0000 7fe9d3373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:02.144 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:02.144+0000 7fe9d3373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:02.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:02.145+0000 7fe9d3373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:02.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:02.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:05:02.666 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:02.666+0000 7fe9d3373640 -1 Falling back to public interface 2024-04-01T06:05:02.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:03.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:03.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:03.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:05:03.686 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:05:03.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:03.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:05:03.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:05:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:05:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:04.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:05:05.008 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:05.008+0000 7fe9d3373640 -1 osd.1 36 log_to_monitors true 2024-04-01T06:05:05.286 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:06.284 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:05:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:05:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:06.285 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1713191036,v1:127.0.0.1:6811/1713191036] [v2:127.0.0.1:6812/1713191036,v1:127.0.0.1:6813/1713191036] exists,up 7ea6b8c0-0c98-4eaf-828a-f43f122c336a 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-04-01T06:05:06.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:05:06.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:05:06.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.2 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:05:06.583 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T06:05:06.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T06:05:06.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:05:06.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:05:06.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:05:06.605 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:06.605+0000 7fd03c624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:06.607 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:06.607+0000 7fd03c624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:06.608 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:06.608+0000 7fd03c624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:06.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:05:07.123 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:07.123+0000 7fd03c624640 -1 Falling back to public interface 2024-04-01T06:05:07.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:08.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:05:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:05:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:08.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:05:08.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:09.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:09.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:09.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:05:09.441 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:05:09.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:09.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:05:09.499 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:09.499+0000 7fd03c624640 -1 osd.2 38 log_to_monitors true 2024-04-01T06:05:09.801 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:10.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:10.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:10.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:05:10.803 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:05:10.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:10.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:05:11.248 INFO:tasks.workunit.client.0.smithi067.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/275689778,v1:127.0.0.1:6819/275689778] [v2:127.0.0.1:6820/275689778,v1:127.0.0.1:6821/275689778] exists,up 94c73cd5-ccc6-4d3b-b476-f670fdf32d41 2024-04-01T06:05:11.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:05:11.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:05:11.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:05:11.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-04-01T06:05:11.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T06:05:11.688 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' removed 2024-04-01T06:05:11.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T06:05:11.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T06:05:12.287 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T06:05:12.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:05:13.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T06:05:14.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-04-01T06:05:14.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:05:14.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:05:14.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:05:14.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:05:14.731 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:05:14.731 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:05:14.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:05:14.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:05:14.731 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:05:14.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:05:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:05:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:05:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:05:14.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:05:14.832 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:05:15.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:05:15.116 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:05:15.117 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:05:15.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:05:15.117 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:05:15.117 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:05:15.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659140 2024-04-01T06:05:15.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659140 2024-04-01T06:05:15.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140' 2024-04-01T06:05:15.221 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:05:15.221 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:05:15.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561028 2024-04-01T06:05:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561028 2024-04-01T06:05:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561028' 2024-04-01T06:05:15.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:05:15.327 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:05:15.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-04-01T06:05:15.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-04-01T06:05:15.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561028 2-201863462915' 2024-04-01T06:05:15.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:05:15.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659140 2024-04-01T06:05:15.433 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:05:15.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:05:15.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659140 2024-04-01T06:05:15.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:05:15.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659140 2024-04-01T06:05:15.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-04-01T06:05:15.436 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 176093659140 2024-04-01T06:05:15.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:05:15.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659140 2024-04-01T06:05:15.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:05:16.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:05:16.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:05:17.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659141 -lt 176093659140 2024-04-01T06:05:17.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:05:17.013 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561028 2024-04-01T06:05:17.013 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:05:17.014 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:05:17.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561028 2024-04-01T06:05:17.015 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:05:17.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561028 2024-04-01T06:05:17.016 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561028' 2024-04-01T06:05:17.016 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 188978561028 2024-04-01T06:05:17.017 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:05:17.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561028 2024-04-01T06:05:17.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:05:17.329 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-04-01T06:05:17.330 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:05:17.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:05:17.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-04-01T06:05:17.331 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:05:17.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-04-01T06:05:17.333 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 201863462915 2024-04-01T06:05:17.333 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-04-01T06:05:17.333 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:05:17.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-04-01T06:05:17.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:05:17.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:05:17.658 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:05:18.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:05:18.018 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:05:18.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T06:05:18.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:05:18.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:05:18.293 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-04-01T06:05:18.651 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:05:18.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:05:19.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T06:04:42.793629+0000 2024-04-01T06:05:19.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T06:05:19.481 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T06:04:42.793629+0000 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:04:42.793629+0000 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:05:19.493 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:05:19.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:05:19.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:05:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:04:42.793629+0000 '>' 2024-04-01T06:04:42.793629+0000 2024-04-01T06:05:19.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:05:20.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:05:20.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:05:20.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:05:20.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:05:20.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:05:20.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:05:20.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:05:21.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:05:19.547248+0000 '>' 2024-04-01T06:04:42.793629+0000 2024-04-01T06:05:21.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:05:21.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-04-01T06:05:21.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-04-01T06:05:21.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-04-01T06:05:21.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-04-01T06:05:21.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-04-01T06:05:21.070 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:05:21.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:05:21.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:05:21.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:05:21.180 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:05:21.287 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:05:21.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:05:21.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:05:21.289 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:05:21.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:05:21.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:05:21.290 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:05:21.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:05:21.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:05:21.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:05:21.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:05:21.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:05:21.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:05:21.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:05:21.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:05:21.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:21.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:21.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:05:21.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:05:21.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:05:21.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:05:21.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:05:21.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:05:21.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:05:21.315 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:05:21.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:05:21.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:05:21.316 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:05:21.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:05:21.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:05:21.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:05:21.318 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:05:21.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:05:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:05:21.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:05:21.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:05:21.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:21.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:21.321 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:05:21.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:05:21.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:05:21.322 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T06:05:21.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T06:05:21.323 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:21.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:21.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T06:05:21.326 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T06:05:21.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T06:05:21.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T06:05:21.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T06:05:21.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T06:05:21.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T06:05:21.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T06:05:21.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T06:05:21.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T06:05:21.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T06:05:21.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:05:21.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:05:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:05:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:21.692 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:21.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T06:05:21.885 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:05:21.886 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:05:21.886 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:05:21.886 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:05:21.887 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:21.887 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:21.887 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:05:21.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:05:21.887 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:05:21.973 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:05:21.975 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:05:21.975 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:21.975 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:21.975 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:05:21.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:05:21.975 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T06:05:22.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-04-01T06:05:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T06:05:22.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T06:05:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T06:05:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T06:05:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T06:05:22.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T06:05:22.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T06:05:22.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:05:22.870 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:05:22.870 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:05:22.870 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:22.871 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:22.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:22.871 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:05:22.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:05:22.900 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-04-01T06:05:22.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:05:22.904 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:05:22.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:05:22.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba 2024-04-01T06:05:22.907 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba 2024-04-01T06:05:22.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba' 2024-04-01T06:05:22.907 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:05:22.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiTgpmqm0BNxAAyQWAf+yLtk2ug/ujz9RHBw== 2024-04-01T06:05:22.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiTgpmqm0BNxAAyQWAf+yLtk2ug/ujz9RHBw=="}' 2024-04-01T06:05:22.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba -i td/osd-scrub-repair/0/new.json 2024-04-01T06:05:23.133 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:05:23.143 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T06:05:23.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiTgpmqm0BNxAAyQWAf+yLtk2ug/ujz9RHBw== --osd-uuid a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba 2024-04-01T06:05:23.172 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.171+0000 7f4611887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:23.184 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.183+0000 7f4611887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:23.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.184+0000 7f4611887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:23.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.184+0000 7f4611887640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:05:23.185 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.184+0000 7f4611887640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:05:23.186 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.184+0000 7f4611887640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:05:23.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:23.275+0000 7f4611887640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T06:05:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T06:05:28.832 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:05:28.833 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T06:05:28.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T06:05:28.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:05:29.202 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:05:29.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T06:05:29.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:05:29.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:05:29.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:05:29.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:05:29.228 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:29.228+0000 7f8d9ee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:29.259 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:29.259+0000 7f8d9ee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:29.260 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:29.260+0000 7f8d9ee0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:29.487 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:05:29.488 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:05:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:29.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:29.776 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:29.776+0000 7f8d9ee0f640 -1 Falling back to public interface 2024-04-01T06:05:29.790 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:30.791 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:30.792 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:05:30.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:05:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:30.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:31.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:32.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:32.084 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:05:32.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:32.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:05:32.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:32.085 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:32.117 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:32.116+0000 7f8d9ee0f640 -1 osd.0 0 log_to_monitors true 2024-04-01T06:05:32.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:33.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:33.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:33.544 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:05:33.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:05:33.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:33.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:33.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:34.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:34.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:34.867 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:05:34.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:05:34.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:34.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:35.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:36.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:36.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:36.161 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:05:36.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:05:36.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:36.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:36.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:37.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:37.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:37.453 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:05:37.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:05:37.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:37.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:37.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:38.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:38.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:38.866 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:05:38.866 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:05:38.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:38.867 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:39.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:40.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:40.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:40.162 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:05:40.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:05:40.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:40.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:40.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:41.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:41.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:41.457 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:05:41.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:05:41.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:41.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:41.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:42.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:42.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:42.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:05:42.751 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:05:42.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:42.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:43.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:44.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:44.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:44.055 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:05:44.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:05:44.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:44.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:44.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:45.410 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:45.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:45.411 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:05:45.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:05:45.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:45.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:45.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:46.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:46.926 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:05:46.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:46.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:05:46.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:46.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:47.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:48.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:48.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:48.822 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:05:48.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:05:48.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:48.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:49.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:50.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:50.197 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:05:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:05:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:50.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:50.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:51.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:51.784 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:51.785 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:05:51.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:05:51.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:51.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:52.380 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:53.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:53.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:53.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:05:53.382 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:05:53.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:53.386 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:53.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:05:54.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:05:54.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:05:54.899 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:05:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:05:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:05:54.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:05:55.220 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3924322485,v1:127.0.0.1:6803/3924322485] [v2:127.0.0.1:6804/3924322485,v1:127.0.0.1:6805/3924322485] exists,up a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba 2024-04-01T06:05:55.224 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:05:55.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:05:55.226 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:05:55.227 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:05:55.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:05:55.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:05:55.228 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:05:55.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0abd9e52-7613-45ca-a2c2-500471f8902b 2024-04-01T06:05:55.228 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 0abd9e52-7613-45ca-a2c2-500471f8902b 2024-04-01T06:05:55.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0abd9e52-7613-45ca-a2c2-500471f8902b' 2024-04-01T06:05:55.229 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:05:55.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDTgpmB/h6DhAAE/wUTR2tJKBXkrBbfD4XzQ== 2024-04-01T06:05:55.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDTgpmB/h6DhAAE/wUTR2tJKBXkrBbfD4XzQ=="}' 2024-04-01T06:05:55.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0abd9e52-7613-45ca-a2c2-500471f8902b -i td/osd-scrub-repair/1/new.json 2024-04-01T06:05:55.617 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:05:55.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T06:05:55.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDTgpmB/h6DhAAE/wUTR2tJKBXkrBbfD4XzQ== --osd-uuid 0abd9e52-7613-45ca-a2c2-500471f8902b 2024-04-01T06:05:55.657 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.657+0000 7f148be1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:55.693 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.693+0000 7f148be1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:55.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.694+0000 7f148be1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:05:55.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.694+0000 7f148be1b640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:05:55.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.694+0000 7f148be1b640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:05:55.695 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.694+0000 7f148be1b640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:05:55.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:05:55.768+0000 7f148be1b640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T06:06:01.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T06:06:01.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:06:01.608 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T06:06:01.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T06:06:01.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:06:01.992 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:06:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T06:06:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:06:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:06:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:06:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:06:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:02.018+0000 7f55f040e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:02.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:02.049+0000 7f55f040e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:02.052 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:02.050+0000 7f55f040e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:02.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:06:02.280 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:06:02.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:02.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:02.566 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:02.565+0000 7f55f040e640 -1 Falling back to public interface 2024-04-01T06:06:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:03.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:03.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:03.587 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:06:03.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:06:03.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:03.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:03.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:04.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:04.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:04.880 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:06:04.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:06:04.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:04.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:04.901 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:04.900+0000 7f55f040e640 -1 osd.1 0 log_to_monitors true 2024-04-01T06:06:05.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:06.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:06.201 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:06.202 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:06:06.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:06:06.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:06.202 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:06.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:07.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:07.498 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:06:07.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:07.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:06:07.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:07.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:07.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:08.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:08.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:08.896 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:06:08.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:06:08.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:08.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:09.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:10.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:10.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:10.191 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:06:10.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:06:10.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:10.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:10.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:11.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:11.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:11.495 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:06:11.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:06:11.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:11.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:11.828 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:12.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:12.830 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:12.830 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:06:12.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:06:12.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:12.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:13.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:14.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:14.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:14.151 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:06:14.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:06:14.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:14.152 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:14.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:15.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:15.481 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:15.482 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:06:15.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:06:15.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:15.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:15.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:16.910 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:16.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:16.911 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:06:16.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:06:16.911 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:16.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:17.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:18.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:18.206 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:18.207 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:06:18.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:06:18.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:18.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:18.803 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:19.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:19.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:19.805 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:06:19.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:06:19.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:19.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:20.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:21.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:21.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:21.441 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:06:21.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:06:21.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:21.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:21.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:22.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:22.783 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:06:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:06:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:22.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:23.241 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:24.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:24.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:24.242 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:06:24.243 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:06:24.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:24.243 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:24.721 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:25.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:25.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:25.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:06:25.723 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:06:25.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:25.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:26.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:27.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:27.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:27.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:06:27.022 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:06:27.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:27.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:27.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:28.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:28.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:28.458 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T06:06:28.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T06:06:28.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:28.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:06:28.751 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/55284421,v1:127.0.0.1:6811/55284421] [v2:127.0.0.1:6812/55284421,v1:127.0.0.1:6813/55284421] exists,up 0abd9e52-7613-45ca-a2c2-500471f8902b 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:06:28.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:06:28.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:06:28.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61043024-fd63-461c-8ff5-e74e0b3ceb4d 2024-04-01T06:06:28.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 61043024-fd63-461c-8ff5-e74e0b3ceb4d' 2024-04-01T06:06:28.759 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 61043024-fd63-461c-8ff5-e74e0b3ceb4d 2024-04-01T06:06:28.759 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:06:28.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkTgpmgXoXLhAAt2xhGOtEYHEQ8yB32tJk8Q== 2024-04-01T06:06:28.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkTgpmgXoXLhAAt2xhGOtEYHEQ8yB32tJk8Q=="}' 2024-04-01T06:06:28.774 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61043024-fd63-461c-8ff5-e74e0b3ceb4d -i td/osd-scrub-repair/2/new.json 2024-04-01T06:06:29.285 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:06:29.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T06:06:29.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkTgpmgXoXLhAAt2xhGOtEYHEQ8yB32tJk8Q== --osd-uuid 61043024-fd63-461c-8ff5-e74e0b3ceb4d 2024-04-01T06:06:29.325 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.325+0000 7f0ff26c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:29.361 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.361+0000 7f0ff26c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:29.362 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.362+0000 7f0ff26c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:29.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.362+0000 7f0ff26c6640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:06:29.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.362+0000 7f0ff26c6640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:06:29.363 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.362+0000 7f0ff26c6640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:06:29.424 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:29.424+0000 7f0ff26c6640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T06:06:35.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T06:06:35.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:06:35.431 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T06:06:35.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T06:06:35.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:06:35.802 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T06:06:35.803 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T06:06:35.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:06:35.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:06:35.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:06:35.804 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:06:35.829 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:35.828+0000 7fbd85410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:35.831 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:35.831+0000 7fbd85410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:35.832 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:35.832+0000 7fbd85410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:06:36.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T06:06:36.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:06:36.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T06:06:36.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:06:36.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:06:36.114 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:36.115 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:06:36.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:06:36.115 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:36.116 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:36.359 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:36.359+0000 7fbd85410640 -1 Falling back to public interface 2024-04-01T06:06:36.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:37.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:37.402 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:06:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:06:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:37.403 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:37.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:38.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:38.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:38.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:06:38.697 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:06:38.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:38.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:38.711 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:06:38.711+0000 7fbd85410640 -1 osd.2 0 log_to_monitors true 2024-04-01T06:06:39.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:40.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:40.070 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:06:40.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:40.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:06:40.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:40.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:40.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:41.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:41.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:41.417 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:06:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:06:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:41.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:41.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:42.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:42.744 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:42.745 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:06:42.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:06:42.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:42.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:43.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:44.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:44.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:44.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:06:44.070 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:06:44.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:44.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:44.414 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:45.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:45.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:45.416 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:06:45.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:06:45.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:45.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:45.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:46.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:46.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:46.756 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:06:46.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:06:46.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:46.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:47.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:48.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:48.054 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:06:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:06:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:48.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:48.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:49.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:49.440 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:49.441 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:06:49.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:06:49.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:49.441 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:49.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:50.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:50.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:50.766 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:06:50.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:06:50.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:50.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:51.105 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:52.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:52.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:52.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:06:52.107 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:06:52.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:52.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:52.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:53.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:53.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:53.598 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:06:53.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:06:53.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:53.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:54.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:55.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:55.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:55.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:06:55.029 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:06:55.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:55.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:55.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:56.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:56.558 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:06:56.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:06:56.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:56.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:57.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:58.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:58.113 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:06:58.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:06:58.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:58.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:06:58.694 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:06:59.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:06:59.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:06:59.696 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:06:59.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:06:59.696 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:06:59.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:07:00.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:01.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:01.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:01.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:07:01.139 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:07:01.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:01.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:07:01.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:02.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T06:07:02.472 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T06:07:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:07:02.766 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1193616377,v1:127.0.0.1:6819/1193616377] [v2:127.0.0.1:6820/1193616377,v1:127.0.0.1:6821/1193616377] exists,up 61043024-fd63-461c-8ff5-e74e0b3ceb4d 2024-04-01T06:07:02.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:07:02.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:07:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:07:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-04-01T06:07:02.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T06:07:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T06:07:03.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T06:07:03.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T06:07:03.452 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T06:07:03.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:07:04.465 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T06:07:06.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-04-01T06:07:06.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:07:06.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:07:06.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:07:06.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:07:06.626 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:07:06.627 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:07:06.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:07:06.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:07:06.627 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:07:06.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:07:06.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:07:06.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:07:06.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:07:06.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:07:06.728 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:07:07.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:07:07.012 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:07.012 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:07:07.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:07:07.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:07.013 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:07:07.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-04-01T06:07:07.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-04-01T06:07:07.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-04-01T06:07:07.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:07.118 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:07:07.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-04-01T06:07:07.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-04-01T06:07:07.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672973' 2024-04-01T06:07:07.223 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:07.223 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:07:07.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-04-01T06:07:07.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-04-01T06:07:07.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672973 2-64424509446' 2024-04-01T06:07:07.335 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:07.336 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-04-01T06:07:07.336 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:07.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:07:07.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-04-01T06:07:07.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:07.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-04-01T06:07:07.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-01T06:07:07.339 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836500 2024-04-01T06:07:07.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:07:07.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-04-01T06:07:07.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:07:08.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:07:08.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:07:08.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-04-01T06:07:08.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:07:09.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-01T06:07:09.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:07:10.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-04-01T06:07:10.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:10.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-04-01T06:07:10.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:10.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:07:10.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-04-01T06:07:10.216 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:10.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-04-01T06:07:10.217 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672973 2024-04-01T06:07:10.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-04-01T06:07:10.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:07:10.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672973 2024-04-01T06:07:10.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:10.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-04-01T06:07:10.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:10.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:07:10.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-04-01T06:07:10.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:10.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-04-01T06:07:10.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-04-01T06:07:10.505 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509446 2024-04-01T06:07:10.505 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:07:10.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-04-01T06:07:10.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:07:10.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:07:10.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:07:11.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T06:07:11.176 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:07:11.176 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:07:11.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:07:11.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:07:11.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:07:11.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:07:11.177 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:07:11.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T06:07:11.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:07:11.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:07:11.464 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:07:11.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T06:07:11.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:07:11.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:07:11.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-04-01T06:07:11.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-04-01T06:07:11.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T06:07:11.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-04-01T06:07:11.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T06:07:11.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-01T06:07:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-04-01T06:07:12.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-04-01T06:07:12.930 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'ecpool' created 2024-04-01T06:07:12.942 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:07:13.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-04-01T06:07:13.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-04-01T06:07:14.325 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 2 allow_ec_overwrites to true 2024-04-01T06:07:14.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T06:07:14.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:07:14.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:07:14.337 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:07:14.337 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:07:14.338 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:07:14.338 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:07:14.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:07:14.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:07:14.338 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:07:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:07:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:07:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:07:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:07:14.428 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:07:14.428 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:07:14.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:07:14.710 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:14.711 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:07:14.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:07:14.711 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:14.711 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:07:14.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-04-01T06:07:14.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-04-01T06:07:14.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-04-01T06:07:14.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:14.816 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:07:14.950 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-04-01T06:07:14.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-04-01T06:07:14.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672976' 2024-04-01T06:07:14.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:14.951 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:07:15.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-04-01T06:07:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-04-01T06:07:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672976 2-64424509448' 2024-04-01T06:07:15.057 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:15.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-04-01T06:07:15.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:15.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:07:15.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-04-01T06:07:15.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-04-01T06:07:15.060 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-01T06:07:15.060 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836503 2024-04-01T06:07:15.061 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:07:15.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-04-01T06:07:15.342 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:15.343 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-04-01T06:07:15.343 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:15.344 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:07:15.344 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-04-01T06:07:15.344 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:15.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-04-01T06:07:15.346 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-04-01T06:07:15.346 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672976 2024-04-01T06:07:15.347 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:07:15.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-04-01T06:07:15.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:15.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-04-01T06:07:15.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:15.654 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:07:15.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-04-01T06:07:15.655 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:15.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-04-01T06:07:15.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-01T06:07:15.657 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509448 2024-04-01T06:07:15.657 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:07:15.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509448 2024-04-01T06:07:15.943 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:07:16.944 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:07:16.944 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:07:17.238 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-04-01T06:07:17.238 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:07:17.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:07:17.239 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:07:17.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:07:17.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:07:17.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T06:07:17.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:07:17.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:07:17.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:07:18.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T06:07:18.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T06:07:18.611 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T06:07:18.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T06:07:19.089 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:07:19.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:07:19.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:07:19.100 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:07:19.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-04-01T06:07:19.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T06:07:19.381 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-04-01T06:07:19.381 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-04-01T06:07:19.381 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:07:19.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-04-01T06:07:19.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-04-01T06:07:19.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-04-01T06:07:20.110 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is set 2024-04-01T06:07:20.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-04-01T06:07:20.503 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:07:20.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:07:20.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:07:20.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:07:20.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-04-01T06:07:20.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-04-01T06:07:20.741 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-04-01T06:07:20.742 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-04-01T06:07:20.742 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:0' 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-04-01T06:07:21.082 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-04-01T06:07:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:07:21.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:07:21.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:07:21.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:07:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:07:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:07:21.084 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:07:21.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:07:21.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:07:21.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:07:21.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:07:21.393 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:07:21.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:07:21.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:07:21.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:07:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:07:23.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:07:23.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-04-01T06:07:23.008 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-04-01T06:07:24.195 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-04-01T06:07:24.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-04-01T06:07:24.710 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-04-01T06:07:25.824 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-04-01T06:07:26.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-04-01T06:07:26.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-04-01T06:07:27.109 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-04-01T06:07:27.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-04-01T06:07:27.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-04-01T06:07:28.898 INFO:tasks.workunit.client.0.smithi067.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-04-01T06:07:29.413 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-04-01T06:07:29.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:07:29.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-04-01T06:07:29.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:07:29.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:07:29.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:07:29.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:07:29.418 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:07:29.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:07:29.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:07:29.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:07:29.420 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T06:07:29.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:07:29.421 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:07:29.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:07:29.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:07:29.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:07:29.421 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:07:29.447 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:29.446+0000 7f3b7420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:29.449 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:29.448+0000 7f3b7420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:29.450 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:29.449+0000 7f3b7420f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:29.713 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:07:29.714 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:29.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:07:29.966 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:29.965+0000 7f3b7420f640 -1 Falling back to public interface 2024-04-01T06:07:30.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:31.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:31.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:31.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:07:31.006 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:31.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:31.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:07:31.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:32.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:32.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:32.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:07:32.310 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:07:32.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:32.310 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:07:32.404 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:32.403+0000 7f3b7420f640 -1 osd.0 35 log_to_monitors true 2024-04-01T06:07:32.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:33.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:33.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:33.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:07:33.671 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:07:33.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:33.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/1451260423,v1:127.0.0.1:6803/1451260423] [v2:127.0.0.1:6804/1451260423,v1:127.0.0.1:6805/1451260423] exists,up a0fe4046-0f55-4ce2-8ed2-d5fabb56e5ba 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:07:33.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:07:33.963 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:07:33.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:07:33.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:07:33.965 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.1 2024-04-01T06:07:33.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:07:33.965 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:07:33.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:07:33.965 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:07:33.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:07:33.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:07:33.988 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:33.986+0000 7f6813425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:33.992 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:33.991+0000 7f6813425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:33.994 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:33.993+0000 7f6813425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:34.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:34.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:07:34.510 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:34.509+0000 7f6813425640 -1 Falling back to public interface 2024-04-01T06:07:34.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:07:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:35.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:07:35.842 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:36.765 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:36.764+0000 7f6813425640 -1 osd.1 36 log_to_monitors true 2024-04-01T06:07:36.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:36.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:36.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:07:36.843 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:07:36.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:36.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:07:37.166 INFO:tasks.workunit.client.0.smithi067.stderr:osd.1 up in weight 1 up_from 44 up_thru 36 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1228450431,v1:127.0.0.1:6811/1228450431] [v2:127.0.0.1:6812/1228450431,v1:127.0.0.1:6813/1228450431] exists,up 0abd9e52-7613-45ca-a2c2-500471f8902b 2024-04-01T06:07:37.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:07:37.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:07:37.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T06:07:37.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:07:37.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.2 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:07:37.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:07:37.192 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:37.191+0000 7f3682b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:37.193 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:37.192+0000 7f3682b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:37.194 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:37.193+0000 7f3682b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:37.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:07:37.599 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:07:37.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:37.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:07:37.714 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:37.714+0000 7f3682b06640 -1 Falling back to public interface 2024-04-01T06:07:37.978 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:38.979 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:07:38.980 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:38.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:38.980 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:07:39.272 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:07:40.204 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:40.204+0000 7f3682b06640 -1 osd.2 38 log_to_monitors true 2024-04-01T06:07:40.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:07:40.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:40.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:07:40.275 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:07:40.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:40.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:07:40.632 INFO:tasks.workunit.client.0.smithi067.stderr:osd.2 up in weight 1 up_from 47 up_thru 37 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/1828253752,v1:127.0.0.1:6819/1828253752] [v2:127.0.0.1:6820/1828253752,v1:127.0.0.1:6821/1828253752] exists,up 61043024-fd63-461c-8ff5-e74e0b3ceb4d 2024-04-01T06:07:40.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:07:40.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:07:40.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:07:40.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-04-01T06:07:40.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T06:07:41.686 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' removed 2024-04-01T06:07:41.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T06:07:41.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T06:07:42.138 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T06:07:42.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:07:43.151 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T06:07:44.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-04-01T06:07:44.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:07:44.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:07:44.966 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:07:44.966 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:07:44.967 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:07:44.967 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:07:44.967 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:07:44.967 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:07:44.967 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:07:45.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:07:45.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:07:45.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:07:45.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:07:45.068 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:07:45.068 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:07:45.350 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:07:45.350 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:07:45.350 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:07:45.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:07:45.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:45.351 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:07:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659140 2024-04-01T06:07:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659140 2024-04-01T06:07:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140' 2024-04-01T06:07:45.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:45.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:07:45.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-04-01T06:07:45.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-04-01T06:07:45.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-04-01T06:07:45.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:07:45.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:07:45.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-04-01T06:07:45.665 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-04-01T06:07:45.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-201863462915' 2024-04-01T06:07:45.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:45.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659140 2024-04-01T06:07:45.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:45.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:07:45.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659140 2024-04-01T06:07:45.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:45.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659140 2024-04-01T06:07:45.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-04-01T06:07:45.669 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 176093659140 2024-04-01T06:07:45.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:07:45.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659140 2024-04-01T06:07:45.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:07:46.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:07:46.953 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:07:47.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659140 -lt 176093659140 2024-04-01T06:07:47.252 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:47.252 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-04-01T06:07:47.253 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:47.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:07:47.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-04-01T06:07:47.254 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:47.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-04-01T06:07:47.256 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 188978561027 2024-04-01T06:07:47.256 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-04-01T06:07:47.256 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:07:47.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561027 2024-04-01T06:07:47.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:07:47.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-04-01T06:07:47.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:07:47.543 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:07:47.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-04-01T06:07:47.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:07:47.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-04-01T06:07:47.545 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 201863462915 2024-04-01T06:07:47.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-04-01T06:07:47.546 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:07:47.827 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-04-01T06:07:47.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:07:47.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:07:47.828 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:07:48.207 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:07:48.547 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T06:07:48.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:07:48.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:07:48.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-04-01T06:07:48.901 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:07:49.188 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:07:49.189 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:07:49.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T06:07:12.816812+0000 2024-04-01T06:07:49.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-04-01T06:07:49.764 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-04-01T06:07:12.816812+0000 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:07:12.816812+0000 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:07:49.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:07:49.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:07:49.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:07:49.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:07:49.777 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:07:50.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:07:12.816812+0000 '>' 2024-04-01T06:07:12.816812+0000 2024-04-01T06:07:50.054 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:07:51.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:07:51.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:07:51.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:07:51.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:07:51.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:07:51.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:07:51.057 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:07:51.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:07:50.203136+0000 '>' 2024-04-01T06:07:12.816812+0000 2024-04-01T06:07:51.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:07:51.373 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-04-01T06:07:51.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-04-01T06:07:51.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-04-01T06:07:51.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-04-01T06:07:51.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-04-01T06:07:51.374 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:07:51.482 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:07:51.483 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:07:51.483 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:07:51.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:07:51.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:07:51.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:07:51.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:07:51.591 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:07:51.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:07:51.592 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:07:51.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:07:51.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:07:51.593 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:07:51.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:07:51.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:07:51.594 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:07:51.595 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:07:51.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:07:51.596 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:07:51.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:07:51.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:07:51.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:51.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:51.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:07:51.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:07:51.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:07:51.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:07:51.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:07:51.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:07:51.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:07:51.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:07:51.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:07:51.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:07:51.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:07:51.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:07:51.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:07:51.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:07:51.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:07:51.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:07:51.621 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:07:51.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:07:51.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:07:51.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:07:51.624 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:07:51.624 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:51.624 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:51.624 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:07:51.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:07:51.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:07:51.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T06:07:51.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T06:07:51.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:51.626 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:51.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T06:07:51.629 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T06:07:51.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T06:07:51.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T06:07:51.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T06:07:51.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-04-01T06:07:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T06:07:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:07:52.001 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:07:52.002 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:07:52.002 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:52.002 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:52.002 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:52.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-04-01T06:07:52.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T06:07:52.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T06:07:52.185 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:07:52.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:07:52.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T06:07:52.186 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:07:52.187 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:07:52.270 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:07:52.271 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:07:52.271 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:52.271 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:52.271 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:07:52.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:07:52.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T06:07:52.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-04-01T06:07:52.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T06:07:52.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T06:07:52.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T06:07:52.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T06:07:52.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T06:07:52.352 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T06:07:53.162 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:53.163 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:07:53.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:07:53.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-04-01T06:07:53.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-04-01T06:07:53.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:07:53.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:07:53.196 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71eb9956-8b5d-4c03-bcef-092a96f7b97e 2024-04-01T06:07:53.197 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 71eb9956-8b5d-4c03-bcef-092a96f7b97e' 2024-04-01T06:07:53.198 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 71eb9956-8b5d-4c03-bcef-092a96f7b97e 2024-04-01T06:07:53.198 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:07:53.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5Twpmwva0DBAAPrGO57UBBMRBl8E5Bjj/Lg== 2024-04-01T06:07:53.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5Twpmwva0DBAAPrGO57UBBMRBl8E5Bjj/Lg=="}' 2024-04-01T06:07:53.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71eb9956-8b5d-4c03-bcef-092a96f7b97e -i td/osd-scrub-repair/0/new.json 2024-04-01T06:07:53.426 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:07:53.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T06:07:53.437 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQA5Twpmwva0DBAAPrGO57UBBMRBl8E5Bjj/Lg== --osd-uuid 71eb9956-8b5d-4c03-bcef-092a96f7b97e 2024-04-01T06:07:53.467 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.467+0000 7f48a1010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:53.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.477+0000 7f48a1010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:53.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.478+0000 7f48a1010640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:53.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.478+0000 7f48a1010640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:07:53.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.478+0000 7f48a1010640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:07:53.478 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.478+0000 7f48a1010640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:07:53.560 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:53.560+0000 7f48a1010640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T06:07:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T06:07:59.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:07:59.128 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T06:07:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T06:07:59.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:07:59.531 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:07:59.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T06:07:59.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-04-01T06:07:59.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:07:59.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:07:59.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:07:59.559 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:59.559+0000 7fd881b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:59.561 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:59.561+0000 7fd881b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:59.562 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:07:59.562+0000 7fd881b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:07:59.817 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:07:59.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:07:59.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:00.078 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:00.078+0000 7fd881b2a640 -1 Falling back to public interface 2024-04-01T06:08:00.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:01.102 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:01.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:01.103 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:08:01.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:08:01.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:01.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:01.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:02.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:02.397 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:08:02.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:02.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:08:02.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:02.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:02.412+0000 7fd881b2a640 -1 osd.0 0 log_to_monitors true 2024-04-01T06:08:02.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:03.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:03.848 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:08:03.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:03.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:08:03.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:03.849 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:04.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:05.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:05.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:05.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:08:05.171 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:08:05.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:05.171 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:05.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:06.462 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:06.463 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:08:06.463 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:08:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:06.464 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:06.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:07.757 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:07.758 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:08:07.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:07.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:08:07.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:07.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:08.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:09.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:09.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:09.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:08:09.162 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:08:09.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:09.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:09.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:10.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:10.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:10.457 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:08:10.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:08:10.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:10.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:10.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:11.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:11.751 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:08:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:08:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:11.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:12.159 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:13.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:13.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:13.161 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:08:13.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:08:13.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:13.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:13.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:14.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:14.454 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:08:14.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:14.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:08:14.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:14.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:15.182 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:16.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:16.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:16.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:08:16.184 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:08:16.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:16.184 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:16.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:17.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:17.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:17.669 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:08:17.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:08:17.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:17.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:18.429 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:19.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:19.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:19.431 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:08:19.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:08:19.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:19.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:19.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:20.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:20.930 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:08:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:08:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:20.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:21.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:22.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:22.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:22.507 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:08:22.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:08:22.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:22.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:22.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:23.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:23.936 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:08:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:08:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:23.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:08:24.289 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1870524457,v1:127.0.0.1:6803/1870524457] [v2:127.0.0.1:6804/1870524457,v1:127.0.0.1:6805/1870524457] exists,up 71eb9956-8b5d-4c03-bcef-092a96f7b97e 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:08:24.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:08:24.294 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:08:24.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-04-01T06:08:24.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:08:24.296 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:08:24.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a34baf72-703b-4be6-8cb4-d63f5a594315 2024-04-01T06:08:24.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a34baf72-703b-4be6-8cb4-d63f5a594315' 2024-04-01T06:08:24.296 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 a34baf72-703b-4be6-8cb4-d63f5a594315 2024-04-01T06:08:24.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:08:24.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYTwpmgOqKEhAAujmGv4YI5A5ZTxMEP9+faA== 2024-04-01T06:08:24.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYTwpmgOqKEhAAujmGv4YI5A5ZTxMEP9+faA=="}' 2024-04-01T06:08:24.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a34baf72-703b-4be6-8cb4-d63f5a594315 -i td/osd-scrub-repair/1/new.json 2024-04-01T06:08:24.754 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:08:24.765 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T06:08:24.766 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBYTwpmgOqKEhAAujmGv4YI5A5ZTxMEP9+faA== --osd-uuid a34baf72-703b-4be6-8cb4-d63f5a594315 2024-04-01T06:08:24.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.797+0000 7fa5a3414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:08:24.848 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.848+0000 7fa5a3414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:08:24.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.849+0000 7fa5a3414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:08:24.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.849+0000 7fa5a3414640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:08:24.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.849+0000 7fa5a3414640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:08:24.849 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.849+0000 7fa5a3414640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:08:24.923 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:24.923+0000 7fa5a3414640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T06:08:30.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T06:08:30.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:08:30.647 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T06:08:30.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T06:08:30.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:08:31.019 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:08:31.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T06:08:31.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-04-01T06:08:31.019 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:08:31.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:08:31.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:08:31.047 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:31.047+0000 7f177b237640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:08:31.049 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:31.049+0000 7f177b237640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:08:31.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:31.050+0000 7f177b237640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:08:31.305 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:08:31.306 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:08:31.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:31.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:31.577 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:31.577+0000 7f177b237640 -1 Falling back to public interface 2024-04-01T06:08:31.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:32.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:32.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:32.588 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:08:32.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:08:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:32.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:32.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:33.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:08:33.885 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:08:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:33.886 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:33.921 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:08:33.921+0000 7f177b237640 -1 osd.1 0 log_to_monitors true 2024-04-01T06:08:34.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:35.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:35.230 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:35.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:08:35.231 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:08:35.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:35.231 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:35.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:36.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:36.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:36.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:08:36.555 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:08:36.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:36.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:36.876 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:37.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:37.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:37.878 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:08:37.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:08:37.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:37.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:38.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:39.174 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:39.175 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:08:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:08:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:39.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:39.469 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:40.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:40.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:40.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:08:40.471 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:08:40.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:40.472 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:40.776 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:41.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:41.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:41.778 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:08:41.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:08:41.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:41.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:42.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:43.071 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:43.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:43.072 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:08:43.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:08:43.072 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:43.073 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:43.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:44.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:44.442 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:44.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:08:44.443 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:08:44.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:44.443 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:44.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:45.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:45.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:45.823 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:08:45.823 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:08:45.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:45.824 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:46.118 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:47.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:47.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:47.123 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:08:47.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:08:47.123 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:47.124 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:47.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:48.589 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:08:48.590 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:08:48.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:48.590 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:48.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:49.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:49.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:49.897 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:08:49.897 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:08:49.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:49.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:50.602 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:51.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:51.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:51.604 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:08:51.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:08:51.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:51.605 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:52.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:53.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:53.121 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:08:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:08:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:53.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:53.597 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:54.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:54.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:54.598 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:08:54.598 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:08:54.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:54.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:54.891 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:55.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:55.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:55.892 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:08:55.892 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:08:55.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:55.893 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:56.607 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:08:57.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:08:57.608 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:08:57.609 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T06:08:57.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T06:08:57.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:08:57.609 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:08:57.929 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3508742945,v1:127.0.0.1:6811/3508742945] [v2:127.0.0.1:6812/3508742945,v1:127.0.0.1:6813/3508742945] exists,up a34baf72-703b-4be6-8cb4-d63f5a594315 2024-04-01T06:08:57.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:08:57.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:08:57.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:08:57.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-04-01T06:08:57.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-01T06:08:58.208 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' does not exist 2024-04-01T06:08:58.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-01T06:08:58.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-01T06:08:58.601 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'rbd' created 2024-04-01T06:08:58.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:08:59.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-01T06:09:01.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-04-01T06:09:01.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:09:01.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:09:01.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:09:01.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:09:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:09:01.314 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:09:01.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:09:01.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:09:01.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:09:01.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:09:01.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:09:01.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:09:01.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:09:01.415 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:09:01.415 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:09:01.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:09:01.716 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:09:01.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:09:01.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:01.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:09:01.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-04-01T06:09:01.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-04-01T06:09:01.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-04-01T06:09:01.822 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:01.822 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:09:01.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-04-01T06:09:01.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-04-01T06:09:01.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672967' 2024-04-01T06:09:01.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:01.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-04-01T06:09:01.928 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:01.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:09:01.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-04-01T06:09:01.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:01.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-04-01T06:09:01.931 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836493 2024-04-01T06:09:01.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-04-01T06:09:01.931 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-04-01T06:09:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:09:03.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:09:03.214 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:03.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-04-01T06:09:03.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:03.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-04-01T06:09:03.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:03.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:09:03.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-04-01T06:09:03.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:03.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-04-01T06:09:03.511 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672967 2024-04-01T06:09:03.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-04-01T06:09:03.511 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:09:03.800 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-04-01T06:09:03.800 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:09:03.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:09:03.801 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:09:04.158 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:09:04.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-01T06:09:04.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:09:04.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:09:04.436 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:09:04.792 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-01T06:09:04.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:09:04.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:09:04.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-04-01T06:09:04.793 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-04-01T06:09:05.165 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'psr_pool' created 2024-04-01T06:09:05.175 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:09:06.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-04-01T06:09:06.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:09:06.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:09:06.178 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:09:06.179 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:09:06.179 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:09:06.180 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:09:06.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:09:06.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:09:06.180 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:09:06.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:09:06.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:09:06.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:09:06.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:09:06.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:09:06.280 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:09:06.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:09:06.566 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:09:06.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:09:06.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:06.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:09:06.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-04-01T06:09:06.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-04-01T06:09:06.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-04-01T06:09:06.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:06.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:09:06.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-04-01T06:09:06.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-04-01T06:09:06.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672969' 2024-04-01T06:09:06.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:06.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-04-01T06:09:06.778 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:06.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:09:06.779 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-04-01T06:09:06.780 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:06.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-04-01T06:09:06.781 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-04-01T06:09:06.781 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836495 2024-04-01T06:09:06.781 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:07.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-04-01T06:09:07.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:09:08.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:09:08.065 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-04-01T06:09:08.382 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:08.382 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-04-01T06:09:08.382 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:08.383 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:09:08.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-04-01T06:09:08.384 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:08.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-04-01T06:09:08.385 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-01T06:09:08.386 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672969 2024-04-01T06:09:08.386 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:09:08.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-01T06:09:08.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:09:08.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:09:08.674 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:09:09.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T06:09:09.029 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:09:09.029 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:09:09.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:09:09.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:09:09.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:09:09.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:09:09.030 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:09:09.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T06:09:09.309 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:09:09.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:09:09.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:09:09.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T06:09:09.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:09:09.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-04-01T06:09:09.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-04-01T06:09:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:noscrub is unset 2024-04-01T06:09:10.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-04-01T06:09:10.556 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is unset 2024-04-01T06:09:10.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-04-01T06:09:10.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-04-01T06:09:10.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-04-01T06:09:10.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-04-01T06:09:10.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-04-01T06:09:10.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-04-01T06:09:10.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-04-01T06:09:10.819 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T06:09:11.101 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-04-01T06:09:11.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-04-01T06:09:11.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-04-01T06:09:11.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-04-01T06:09:11.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-04-01T06:09:11.102 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:09:11.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:09:11.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:09:11.392 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:09:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:09:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:09:11.392 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:09:11.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:09:11.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T06:09:11.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:09:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:09:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:09:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:09:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:09:11.698 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-04-01T06:09:13.530 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-04-01T06:09:13.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:09:13.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:09:13.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:09:13.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:09:13.615 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-04-01T06:09:13.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:start osd.0 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:13.562+0000 7f4858868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:13.564+0000 7f4858868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:13.618 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:13.566+0000 7f4858868640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:13.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:0 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:13.826 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:14.087 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:14.081+0000 7f4858868640 -1 Falling back to public interface 2024-04-01T06:09:14.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:09:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:09:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:09:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:09:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:15.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:15.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:09:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:09:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:09:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:2 2024-04-01T06:09:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:16.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:16.523 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:16.523+0000 7f4858868640 -1 osd.0 25 log_to_monitors true 2024-04-01T06:09:16.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:09:17.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:09:17.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:17.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:09:17.729 INFO:tasks.workunit.client.0.smithi067.stderr:3 2024-04-01T06:09:17.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:17.730 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:18.103 INFO:tasks.workunit.client.0.smithi067.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/3259231596,v1:127.0.0.1:6803/3259231596] [v2:127.0.0.1:6804/3259231596,v1:127.0.0.1:6805/3259231596] exists,up 71eb9956-8b5d-4c03-bcef-092a96f7b97e 2024-04-01T06:09:18.103 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:09:18.104 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:09:18.105 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:09:18.105 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:09:18.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:09:18.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:09:18.105 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:09:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:09:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:09:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:09:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:09:18.194 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:09:18.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:09:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:09:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:09:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:09:18.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:18.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:09:18.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-04-01T06:09:18.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-04-01T06:09:18.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-04-01T06:09:18.647 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:18.647 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:09:18.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-04-01T06:09:18.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-04-01T06:09:18.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672973' 2024-04-01T06:09:18.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:18.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-04-01T06:09:18.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:18.754 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:09:18.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-04-01T06:09:18.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:18.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-04-01T06:09:18.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-04-01T06:09:18.757 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.0 seq 124554051586 2024-04-01T06:09:18.757 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:19.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-04-01T06:09:19.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:09:20.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:09:20.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:20.357 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-04-01T06:09:20.358 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:20.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-04-01T06:09:20.358 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:20.359 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:09:20.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-04-01T06:09:20.360 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:20.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-04-01T06:09:20.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-04-01T06:09:20.361 INFO:tasks.workunit.client.0.smithi067.stderr:waiting osd.1 seq 42949672973 2024-04-01T06:09:20.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:09:20.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-04-01T06:09:20.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:09:20.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:09:20.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:09:21.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-04-01T06:09:21.004 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:09:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:09:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:09:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:09:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:09:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:09:21.005 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:09:21.283 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-01T06:09:21.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:09:21.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:09:21.284 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:09:21.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-04-01T06:09:21.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:09:21.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:09:21.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-04-01T06:09:21.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-04-01T06:09:21.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-04-01T06:09:21.676 INFO:tasks.workunit.client.0.smithi067.stderr:No scrub information available for pg 2.0 2024-04-01T06:09:21.676 INFO:tasks.workunit.client.0.smithi067.stderr:error 2: (2) No such file or directory 2024-04-01T06:09:21.677 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-04-01T06:09:21.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-04-01T06:09:21.678 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-04-01T06:09:21.678 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-04-01T06:09:21.678 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:21.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:09:21.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:21.679 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-04-01T06:09:21.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:21.991 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-04-01T06:09:22.258 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-04-01T06:09:22.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-04-01T06:09:05.119266+0000 last_deep_scrub_stamp 2024-04-01T06:09:22.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:22.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-04-01T06:09:22.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:05.119266+0000 '>' 2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:22.553 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:23.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:23.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:23.555 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-04-01T06:09:23.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:23.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:09:23.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:23.556 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-04-01T06:09:23.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:05.119266+0000 '>' 2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:23.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:24.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:24.872 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:24.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-04-01T06:09:24.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:24.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:09:24.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:24.873 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-04-01T06:09:25.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:05.119266+0000 '>' 2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:25.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:26.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:26.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:26.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-04-01T06:09:26.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:26.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:09:26.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:26.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-04-01T06:09:26.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:05.119266+0000 '>' 2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:26.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:27.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:27.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:27.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-04-01T06:09:27.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:27.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-04-01T06:09:27.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:27.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-04-01T06:09:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:22.595736+0000 '>' 2024-04-01T06:09:05.119266+0000 2024-04-01T06:09:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:09:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-04-01T06:09:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-04-01T06:09:27.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-04-01T06:09:27.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-04-01T06:09:27.775 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:09:27.776 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:09:28.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:09:28.059 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:09:28.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:09:28.059 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:28.059 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:09:28.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-04-01T06:09:28.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-04-01T06:09:28.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-04-01T06:09:28.164 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:28.164 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:09:28.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-04-01T06:09:28.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-04-01T06:09:28.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672976' 2024-04-01T06:09:28.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:28.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-04-01T06:09:28.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:28.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:09:28.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-04-01T06:09:28.271 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:28.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-04-01T06:09:28.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-04-01T06:09:28.273 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 124554051589 2024-04-01T06:09:28.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:28.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-04-01T06:09:28.555 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:09:29.556 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:09:29.557 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:29.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-04-01T06:09:29.853 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:29.854 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-04-01T06:09:29.854 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:29.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:09:29.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-04-01T06:09:29.855 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:29.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-04-01T06:09:29.857 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-04-01T06:09:29.857 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672976 2024-04-01T06:09:29.857 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:09:30.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-04-01T06:09:30.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-04-01T06:09:30.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:30.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:30.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:30.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:30.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-04-01T06:09:22.595736+0000 2024-04-01T06:09:30.419 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-04-01T06:09:30.518 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:09:30.519 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": false, 2024-04-01T06:09:30.519 INFO:tasks.workunit.client.0.smithi067.stdout: "must": false, 2024-04-01T06:09:30.519 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "2024-03-25T06:07:50.519363+0000" 2024-04-01T06:09:30.519 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:09:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-04-01T06:09:22.595736+0000 2024-04-01T06:09:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T06:09:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:09:22.595736+0000 2024-04-01T06:09:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:09:30.528 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:09:30.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:30.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:09:30.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:30.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:30.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:30.529 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:30.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:22.595736+0000 '>' 2024-04-01T06:09:22.595736+0000 2024-04-01T06:09:30.806 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:31.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:31.807 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:31.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:09:31.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:31.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:31.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:31.808 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:22.595736+0000 '>' 2024-04-01T06:09:22.595736+0000 2024-04-01T06:09:32.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:33.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:33.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:33.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:09:33.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:33.098 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:33.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:33.099 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:33.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:31.529524+0000 '>' 2024-04-01T06:09:22.595736+0000 2024-04-01T06:09:33.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:09:33.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-04-01T06:09:33.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-04-01T06:09:33.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-04-01T06:09:33.391 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-04-01T06:09:33.417 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-04-01T06:09:33.810 INFO:tasks.workunit.client.0.smithi067.stderr:nodeep-scrub is set 2024-04-01T06:09:33.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-04-01T06:09:33.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-04-01T06:09:33.939 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:09:33.939 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:09:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:09:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:09:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:09:34.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:34.257 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:09:34.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-04-01T06:09:34.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-04-01T06:09:34.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-04-01T06:09:34.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:34.362 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:09:34.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-04-01T06:09:34.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-04-01T06:09:34.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-42949672978' 2024-04-01T06:09:34.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:34.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-04-01T06:09:34.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:34.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:09:34.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-04-01T06:09:34.502 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:34.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-04-01T06:09:34.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-04-01T06:09:34.504 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 124554051591 2024-04-01T06:09:34.504 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:34.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-04-01T06:09:34.785 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:09:35.786 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:09:35.786 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-04-01T06:09:36.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:36.080 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-04-01T06:09:36.080 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:36.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:09:36.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-04-01T06:09:36.081 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:36.082 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-04-01T06:09:36.083 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-01T06:09:36.083 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672978 2024-04-01T06:09:36.083 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:09:36.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-01T06:09:36.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-04-01T06:09:41.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-04-01T06:09:41.489 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:09:41.489 INFO:tasks.workunit.client.0.smithi067.stdout: "deep": false, 2024-04-01T06:09:41.489 INFO:tasks.workunit.client.0.smithi067.stdout: "must": false, 2024-04-01T06:09:41.489 INFO:tasks.workunit.client.0.smithi067.stdout: "stamp": "2024-03-25T06:08:01.489751+0000" 2024-04-01T06:09:41.489 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:09:41.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-04-01T06:09:41.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-04-01T06:09:41.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-04-01T06:09:41.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-04-01T06:09:42.501 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-04-01T06:09:42.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-04-01T06:09:42.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-04-01T06:09:42.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-04-01T06:09:42.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-04-01T06:09:42.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-04-01T06:09:42.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-04-01T06:09:42.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-04-01T06:09:42.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:09:42.536 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:09:42.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:09:42.825 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:09:42.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:09:42.825 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:42.825 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:09:42.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051594 2024-04-01T06:09:42.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051594 2024-04-01T06:09:42.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594' 2024-04-01T06:09:42.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:09:42.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:09:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-04-01T06:09:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-04-01T06:09:43.106 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-42949672981' 2024-04-01T06:09:43.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:43.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051594 2024-04-01T06:09:43.107 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:43.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:09:43.108 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051594 2024-04-01T06:09:43.109 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:43.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051594 2024-04-01T06:09:43.110 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 124554051594 2024-04-01T06:09:43.110 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-04-01T06:09:43.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:43.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051594 2024-04-01T06:09:43.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:09:44.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:09:44.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:09:44.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051594 -lt 124554051594 2024-04-01T06:09:44.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:09:44.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-04-01T06:09:44.691 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:09:44.693 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:09:44.693 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-04-01T06:09:44.694 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:09:44.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-04-01T06:09:44.695 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672981 2024-04-01T06:09:44.695 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-01T06:09:44.695 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:09:44.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-04-01T06:09:44.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-04-01T06:09:44.977 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-04-01T06:09:44.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-04-01T06:09:44.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:44.977 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:44.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:44.978 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-03-25T06:08:01.489751+0000 2024-04-01T06:09:45.257 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-04-01T06:09:45.525 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 2.0 on osd.0 to scrub 2024-04-01T06:09:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-03-25T06:08:01.489751+0000 2024-04-01T06:09:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-04-01T06:09:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-25T06:08:01.489751+0000 2024-04-01T06:09:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:09:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:09:45.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:45.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:09:45.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:45.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:45.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:45.538 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:45.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T06:08:01.489751+0000 '>' 2024-03-25T06:08:01.489751+0000 2024-04-01T06:09:45.818 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:46.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:46.819 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:46.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:09:46.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:46.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:46.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:46.820 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:47.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-25T06:08:01.489751+0000 '>' 2024-03-25T06:08:01.489751+0000 2024-04-01T06:09:47.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:09:48.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:09:48.113 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:09:48.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-04-01T06:09:48.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-04-01T06:09:48.113 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:09:48.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:09:48.114 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-04-01T06:09:48.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:09:45.716189+0000 '>' 2024-03-25T06:08:01.489751+0000 2024-04-01T06:09:48.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:09:48.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-04-01T06:09:48.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-04-01T06:09:48.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-04-01T06:09:48.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-04-01T06:09:48.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T06:09:48.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:09:48.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:09:48.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:09:48.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:09:48.434 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:09:48.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:09:48.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:09:48.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:09:48.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:09:48.541 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:09:48.542 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:09:48.542 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:09:48.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:09:48.544 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:09:48.544 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:09:48.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:09:48.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:09:48.545 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:09:48.545 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:09:48.546 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:09:48.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:09:48.548 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:09:48.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:09:48.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:48.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:48.559 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:09:48.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:09:48.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:09:48.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T06:09:48.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T06:09:48.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:09:48.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:09:48.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:09:48.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:09:48.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:09:48.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:09:48.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:09:48.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:09:48.566 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:09:48.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:09:48.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:09:48.567 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:09:48.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:09:48.568 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:09:48.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:09:48.569 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:09:48.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:09:48.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:48.571 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:48.571 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:09:48.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:09:48.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:09:48.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T06:09:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T06:09:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:48.573 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:48.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-04-01T06:09:48.576 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T06:09:48.577 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:09:48.949 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T06:09:49.132 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:09:49.134 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:09:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:09:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:49.135 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:09:49.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:09:49.135 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:09:49.218 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:09:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:09:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:49.219 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:09:49.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:09:49.219 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T06:09:49.299 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-04-01T06:09:49.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T06:09:49.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T06:09:49.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T06:09:49.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T06:09:49.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T06:09:49.300 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:50.110 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:09:50.111 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:09:50.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:09:50.137 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:09:50.138 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-04-01T06:09:50.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:09:50.144 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:09:50.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:09:50.146 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:09:50.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46f3491c-8129-4718-8f79-b2af6ea8e527 2024-04-01T06:09:50.147 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 46f3491c-8129-4718-8f79-b2af6ea8e527' 2024-04-01T06:09:50.147 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 46f3491c-8129-4718-8f79-b2af6ea8e527 2024-04-01T06:09:50.148 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:09:50.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuTwpmzgKrCRAA8PXJY04AnAfmWG/tEMEG4A== 2024-04-01T06:09:50.162 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuTwpmzgKrCRAA8PXJY04AnAfmWG/tEMEG4A=="}' 2024-04-01T06:09:50.163 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46f3491c-8129-4718-8f79-b2af6ea8e527 -i td/osd-scrub-repair/0/new.json 2024-04-01T06:09:50.366 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:09:50.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T06:09:50.376 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCuTwpmzgKrCRAA8PXJY04AnAfmWG/tEMEG4A== --osd-uuid 46f3491c-8129-4718-8f79-b2af6ea8e527 2024-04-01T06:09:50.406 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.406+0000 7f2b240f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:50.416 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.417+0000 7f2b240f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.418+0000 7f2b240f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.418+0000 7f2b240f5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:09:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.418+0000 7f2b240f5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:09:50.418 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.418+0000 7f2b240f5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:09:50.509 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:50.508+0000 7f2b240f5640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T06:09:56.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T06:09:56.056 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:09:56.058 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T06:09:56.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T06:09:56.058 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:09:56.430 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T06:09:56.431 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:09:56.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:09:56.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:09:56.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:09:56.432 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:09:56.459 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:56.458+0000 7f5920e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:56.476 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:56.476+0000 7f5920e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:56.477 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:56.477+0000 7f5920e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:56.716 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:09:56.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:09:56.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:56.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:56.996 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:56.996+0000 7f5920e11640 -1 Falling back to public interface 2024-04-01T06:09:56.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:09:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:09:58.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:58.000 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:09:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:09:58.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:58.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:58.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:09:59.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:09:59.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:09:59.298 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:09:59.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:09:59.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:09:59.298 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:09:59.316 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:09:59.316+0000 7f5920e11640 -1 osd.0 0 log_to_monitors true 2024-04-01T06:09:59.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:10:00.735 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:10:00.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:00.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:01.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:02.065 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:02.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:02.066 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:10:02.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:10:02.066 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:02.067 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:02.360 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:03.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:03.361 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:03.362 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:10:03.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:10:03.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:03.362 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:03.656 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:04.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:04.657 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:04.658 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:10:04.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:10:04.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:04.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:05.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:06.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:06.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:06.036 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:10:06.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:10:06.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:06.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:06.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:07.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:07.330 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:07.330 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:10:07.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:10:07.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:07.331 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:07.625 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:08.626 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:08.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:08.627 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:10:08.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:10:08.627 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:08.628 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:08.920 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:09.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:09.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:09.921 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:10:09.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:10:09.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:09.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:10.213 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:11.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:11.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:11.214 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:10:11.215 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:10:11.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:11.215 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:12.077 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:13.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:13.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:13.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:10:13.079 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:10:13.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:13.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:13.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:14.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:14.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:14.494 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:10:14.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:10:14.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:14.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:15.364 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:16.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:16.366 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:16.366 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:10:16.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:10:16.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:16.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:16.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:17.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:17.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:17.837 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:10:17.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:10:17.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:17.837 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:18.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:19.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:19.438 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:19.438 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:10:19.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:10:19.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:19.439 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:20.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:21.190 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:21.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:21.191 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:10:21.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:10:21.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:21.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:10:21.528 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2409596456,v1:127.0.0.1:6803/2409596456] [v2:127.0.0.1:6804/2409596456,v1:127.0.0.1:6805/2409596456] exists,up 46f3491c-8129-4718-8f79-b2af6ea8e527 2024-04-01T06:10:21.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:10:21.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:10:21.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:10:21.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T06:10:21.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:10:21.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:10:21.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:10:21.534 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:10:21.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89e93dc1-8a84-471e-aec6-f26c7fa037c0 2024-04-01T06:10:21.536 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 89e93dc1-8a84-471e-aec6-f26c7fa037c0 2024-04-01T06:10:21.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89e93dc1-8a84-471e-aec6-f26c7fa037c0' 2024-04-01T06:10:21.537 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:10:21.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNTwpmYZK8IBAAGt4NHSH/Brt3ZKwpNLLf9w== 2024-04-01T06:10:21.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNTwpmYZK8IBAAGt4NHSH/Brt3ZKwpNLLf9w=="}' 2024-04-01T06:10:21.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89e93dc1-8a84-471e-aec6-f26c7fa037c0 -i td/osd-scrub-repair/1/new.json 2024-04-01T06:10:21.885 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:10:21.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T06:10:21.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDNTwpmYZK8IBAAGt4NHSH/Brt3ZKwpNLLf9w== --osd-uuid 89e93dc1-8a84-471e-aec6-f26c7fa037c0 2024-04-01T06:10:21.926 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:21.926+0000 7f709c015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:10:21.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:21.969+0000 7f709c015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:10:21.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:21.970+0000 7f709c015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:10:21.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:21.970+0000 7f709c015640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:10:21.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:21.970+0000 7f709c015640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:10:21.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:21.970+0000 7f709c015640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:10:22.044 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:22.044+0000 7f709c015640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T06:10:28.107 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T06:10:28.108 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:10:28.108 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T06:10:28.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T06:10:28.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:10:28.483 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:10:28.483 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T06:10:28.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:10:28.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:10:28.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:10:28.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:10:28.511 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:28.510+0000 7f9634eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:10:28.513 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:28.512+0000 7f9634eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:10:28.514 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:28.513+0000 7f9634eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:10:28.767 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:10:28.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:28.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:29.030 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:29.029+0000 7f9634eae640 -1 Falling back to public interface 2024-04-01T06:10:29.045 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:30.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:30.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:30.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:10:30.048 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:10:30.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:30.048 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:30.343 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:31.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:31.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:31.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:10:31.345 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:10:31.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:31.345 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:31.458 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:10:31.457+0000 7f9634eae640 -1 osd.1 0 log_to_monitors true 2024-04-01T06:10:31.658 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:32.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:32.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:32.659 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:10:32.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:10:32.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:32.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:32.951 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:33.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:33.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:33.953 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:10:33.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:10:33.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:33.954 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:34.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:35.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:35.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:35.282 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:10:35.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:10:35.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:35.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:35.591 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:36.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:36.592 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:36.593 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:10:36.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:10:36.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:36.593 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:36.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:37.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:37.885 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:37.886 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:10:37.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:10:37.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:37.887 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:38.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:39.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:39.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:10:39.292 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:10:39.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:39.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:39.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:40.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:40.599 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:40.600 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:10:40.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:10:40.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:40.600 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:40.894 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:41.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:41.895 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:41.896 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:10:41.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:10:41.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:41.896 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:42.267 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:43.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:43.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:43.269 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:10:43.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:10:43.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:43.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:44.079 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:45.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:45.080 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:45.081 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:10:45.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:10:45.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:45.081 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:45.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:46.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:46.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:46.883 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:10:46.883 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:10:46.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:46.884 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:47.292 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:48.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:48.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:48.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:10:48.294 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:10:48.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:48.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:48.777 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:49.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:49.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:49.779 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:10:49.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:10:49.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:49.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:50.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:51.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:51.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:51.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:10:51.563 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:10:51.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:51.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:52.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:53.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:53.293 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:53.294 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:10:53.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:10:53.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:53.295 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:53.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:10:54.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:10:54.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:10:54.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:10:54.845 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:10:54.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:10:54.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:10:55.136 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2011291291,v1:127.0.0.1:6811/2011291291] [v2:127.0.0.1:6812/2011291291,v1:127.0.0.1:6813/2011291291] exists,up 89e93dc1-8a84-471e-aec6-f26c7fa037c0 2024-04-01T06:10:55.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:10:55.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:10:55.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:10:55.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-04-01T06:10:55.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-04-01T06:10:55.520 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T06:10:55.532 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:10:56.533 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-04-01T06:10:57.051 INFO:tasks.workunit.client.0.smithi067.stderr:set pool 1 size to 2 2024-04-01T06:10:57.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-04-01T06:10:57.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:10:57.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:10:57.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:10:57.063 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:10:57.064 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:10:57.064 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:10:57.064 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:10:57.064 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:10:57.064 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:10:57.153 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:10:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:10:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:10:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:10:57.154 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:10:57.154 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:10:57.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:10:57.709 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:10:57.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:10:57.709 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:10:57.709 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:10:57.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-04-01T06:10:57.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-04-01T06:10:57.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-04-01T06:10:57.814 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:10:57.814 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:10:57.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-04-01T06:10:57.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-04-01T06:10:57.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672967' 2024-04-01T06:10:57.919 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:10:57.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-04-01T06:10:57.920 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:10:57.921 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:10:57.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-04-01T06:10:57.921 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:10:57.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-04-01T06:10:57.923 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-04-01T06:10:57.923 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836493 2024-04-01T06:10:57.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:10:58.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836493 2024-04-01T06:10:58.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:10:59.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:10:59.206 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:10:59.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-04-01T06:10:59.499 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:10:59.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-04-01T06:10:59.499 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:10:59.500 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:10:59.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-04-01T06:10:59.501 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:10:59.502 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-04-01T06:10:59.502 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672967 2024-04-01T06:10:59.503 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-04-01T06:10:59.503 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:10:59.787 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-04-01T06:10:59.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:10:59.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:10:59.788 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:11:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T06:11:00.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:11:00.140 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:11:00.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:11:00.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:11:00.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:11:00.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:11:00.141 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:11:00.416 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T06:11:00.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:11:00.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:11:00.417 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:11:00.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T06:11:00.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:11:00.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:11:00.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-04-01T06:11:00.768 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-04-01T06:11:00.768 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-04-01T06:11:00.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:00.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:00.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:00.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.181 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.332 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.449 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.809 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:01.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:01.928 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.160 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.420 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.680 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.805 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:02.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:02.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.324 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.475 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.601 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.836 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:03.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:03.952 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:04.069 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:04.070 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:04.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:04.187 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:04.312 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:04.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:04.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-04-01T06:11:04.422 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:11:04.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-04-01T06:11:04.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T06:11:04.548 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-01T06:11:04.548 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-04-01T06:11:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T06:11:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T06:11:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T06:11:04.549 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T06:11:04.830 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T06:11:04.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-04-01T06:11:04.831 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T06:11:05.120 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-04-01T06:11:05.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-04-01T06:11:05.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T06:11:05.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-01T06:11:05.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-04-01T06:11:05.121 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:11:05.405 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-04-01T06:11:05.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-04-01T06:11:05.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T06:11:05.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T06:11:05.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T06:11:05.405 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T06:11:05.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-04-01T06:11:05.689 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:11:05.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:11:05.690 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:11:05.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:11:05.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:11:05.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:11:05.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:11:05.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:11:05.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:11:05.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:11:05.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:11:05.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:11:05.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:11:06.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:11:06.305 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-04-01T06:11:06.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:06.306 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:06.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-04-01T06:11:07.531 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:ff7b1f36:::obj1:head# 2024-04-01T06:11:08.047 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:08.048 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-04-01T06:11:08.049 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:08.050 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-04-01T06:11:09.101 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:104778fc:::obj2:head# 2024-04-01T06:11:09.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:09.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:09.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-04-01T06:11:10.412 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:8dd16f86:::obj3:head# 2024-04-01T06:11:10.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:10.927 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-04-01T06:11:10.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:10.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-04-01T06:11:10.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:10.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:10.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:10.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:10.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:10.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-04-01T06:11:11.757 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:0ee9ae15:::obj4:head# 2024-04-01T06:11:12.273 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:12.273 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-04-01T06:11:12.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:12.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-04-01T06:11:12.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:12.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:12.275 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:12.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:12.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:12.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-04-01T06:11:12.959 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:c52c9666:::obj5:head# 2024-04-01T06:11:13.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:13.477 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:13.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-04-01T06:11:14.152 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:233a42c1:::obj6:head# 2024-04-01T06:11:14.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:14.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:14.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-04-01T06:11:15.439 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:9a09113a:::obj7:head# 2024-04-01T06:11:15.953 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:15.954 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:15.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-04-01T06:11:16.624 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:a557efb1:::obj8:head# 2024-04-01T06:11:17.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:17.139 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-04-01T06:11:17.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:17.142 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-04-01T06:11:17.776 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:fe351e27:::obj9:head# 2024-04-01T06:11:18.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:18.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:18.294 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-04-01T06:11:18.952 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:d1337354:::obj10:head# 2024-04-01T06:11:19.476 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:19.476 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-04-01T06:11:19.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:19.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-04-01T06:11:19.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:19.478 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:19.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:19.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:19.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:19.479 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-04-01T06:11:20.155 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:eeae2a94:::obj11:head# 2024-04-01T06:11:20.922 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:20.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-04-01T06:11:20.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:20.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-04-01T06:11:21.625 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:af5f95cb:::obj12:head# 2024-04-01T06:11:22.390 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:22.391 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-04-01T06:11:22.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:22.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-04-01T06:11:22.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:22.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:22.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:22.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:22.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:22.394 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-04-01T06:11:23.003 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:a61ad63e:::obj13:head# 2024-04-01T06:11:23.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:23.517 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-04-01T06:11:23.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:23.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-04-01T06:11:23.519 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:23.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:23.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:23.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:23.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:23.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-04-01T06:11:24.113 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:e0a44829:::obj14:head# 2024-04-01T06:11:24.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:24.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-04-01T06:11:24.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:24.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-04-01T06:11:25.230 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:ab946124:::obj15:head# 2024-04-01T06:11:25.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:25.749 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-04-01T06:11:25.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:25.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-04-01T06:11:25.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:25.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:25.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:25.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:25.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:25.752 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-04-01T06:11:26.417 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:461f8b5e:::obj16:head# 2024-04-01T06:11:26.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:26.932 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:26.934 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:26.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-04-01T06:11:27.610 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:24fc6e92:::obj17:head# 2024-04-01T06:11:28.125 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:28.125 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-04-01T06:11:28.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:28.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-04-01T06:11:28.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:28.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:28.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:28.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:28.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:28.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-04-01T06:11:28.712 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:b9836a99:::obj18:head# 2024-04-01T06:11:29.226 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:29.226 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-04-01T06:11:29.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-04-01T06:11:29.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-04-01T06:11:29.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:29.228 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:29.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:11:29.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:29.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:29.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-04-01T06:11:29.874 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:00b31b6c:::obj19:head# 2024-04-01T06:11:30.393 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-04-01T06:11:30.394 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:30.396 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-04-01T06:11:31.043 INFO:tasks.workunit.client.0.smithi067.stdout:remove #1:24b5611b:::obj20:head# 2024-04-01T06:11:31.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:11:31.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:11:31.560 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:11:31.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:11:31.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:11:31.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:11:31.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:11:31.563 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:11:31.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:11:31.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:31.591+0000 7fbe4be0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:31.594 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:31.594+0000 7fbe4be0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:31.595 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:31.595+0000 7fbe4be0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:31.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:11:31.859 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:11:31.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:31.859 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:11:32.109 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:32.109+0000 7fbe4be0d640 -1 Falling back to public interface 2024-04-01T06:11:32.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:11:33.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:11:33.140 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:33.140 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:11:33.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:11:33.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:33.141 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:11:33.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:11:34.414 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:34.414+0000 7fbe4be0d640 -1 osd.1 15 log_to_monitors true 2024-04-01T06:11:34.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:11:34.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:34.436 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:11:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:11:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:34.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:11:34.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:11:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:11:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:35.728 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:11:35.728 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:11:35.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:35.729 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:11:36.018 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 18 up_thru 18 down_at 16 last_clean_interval [10,15) [v2:127.0.0.1:6802/3361754376,v1:127.0.0.1:6803/3361754376] [v2:127.0.0.1:6804/3361754376,v1:127.0.0.1:6805/3361754376] exists,up 89e93dc1-8a84-471e-aec6-f26c7fa037c0 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:11:36.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:11:36.023 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:36.024 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:11:36.025 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:11:36.026 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:11:36.026 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:11:36.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:11:36.027 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:11:36.028 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:11:36.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:11:36.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:11:36.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:11:36.028 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:11:36.048 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:36.048+0000 7f902f8c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:36.050 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:36.050+0000 7f902f8c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:36.051 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:36.051+0000 7f902f8c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:36.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:11:36.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:11:36.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:11:36.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:11:36.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:11:36.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:36.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:11:36.308 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:11:36.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:36.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:11:36.565 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:36.565+0000 7f902f8c8640 -1 Falling back to public interface 2024-04-01T06:11:36.585 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:11:37.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:11:37.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:37.587 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:11:37.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:11:37.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:37.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:11:37.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:11:38.817 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:38.816+0000 7f902f8c8640 -1 osd.0 14 log_to_monitors true 2024-04-01T06:11:38.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:11:38.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:11:38.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:11:38.880 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:11:38.880 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:11:38.881 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:11:39.215 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 15 last_clean_interval [5,14) [v2:127.0.0.1:6810/428921533,v1:127.0.0.1:6811/428921533] [v2:127.0.0.1:6812/428921533,v1:127.0.0.1:6813/428921533] exists,up 46f3491c-8129-4718-8f79-b2af6ea8e527 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:11:39.217 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:11:39.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:11:39.218 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:11:39.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:11:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:11:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:11:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:11:39.308 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:11:39.308 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:11:39.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:11:39.616 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:11:39.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:11:39.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:11:39.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:11:39.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313218 2024-04-01T06:11:39.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313218 2024-04-01T06:11:39.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313218' 2024-04-01T06:11:39.731 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:11:39.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:11:39.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411331 2024-04-01T06:11:39.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411331 2024-04-01T06:11:39.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313218 1-77309411331' 2024-04-01T06:11:39.875 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:11:39.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-90194313218 2024-04-01T06:11:39.876 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:11:39.877 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:11:39.877 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-90194313218 2024-04-01T06:11:39.877 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:11:39.878 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313218 2024-04-01T06:11:39.879 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-04-01T06:11:39.879 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 90194313218 2024-04-01T06:11:39.879 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:11:40.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 90194313218 2024-04-01T06:11:40.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:11:41.161 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:11:41.161 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:11:41.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313218 -lt 90194313218 2024-04-01T06:11:41.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:11:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411331 2024-04-01T06:11:41.456 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:11:41.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:11:41.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411331 2024-04-01T06:11:41.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:11:41.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411331 2024-04-01T06:11:41.459 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 77309411331 2024-04-01T06:11:41.460 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411331' 2024-04-01T06:11:41.460 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:11:41.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411331 -lt 77309411331 2024-04-01T06:11:41.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:11:41.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:11:41.746 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:11:42.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T06:11:42.099 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:11:42.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:11:42.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:11:42.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:11:42.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:11:42.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:11:42.100 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:11:42.379 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T06:11:42.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:11:42.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:11:42.380 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-04-01T06:11:42.732 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:11:42.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:11:42.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:11:42.733 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:11:43.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:43.009 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-04-01T06:11:43.281 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 1.0 on osd.1 to repair 2024-04-01T06:11:43.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:11:43.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:11:43.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:10:55.465432+0000 '>' 2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:43.572 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:11:44.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:11:44.573 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:11:44.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:11:44.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:11:44.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:11:44.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:11:44.574 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:11:44.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:10:55.465432+0000 '>' 2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:44.864 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:11:45.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:11:45.865 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:11:45.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:11:45.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:11:45.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:11:45.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:11:45.866 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:11:46.156 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:10:55.465432+0000 '>' 2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:46.157 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:11:47.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:11:47.158 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:11:47.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:11:47.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:11:47.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:11:47.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:11:47.159 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:11:47.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:11:43.491438+0000 '>' 2024-04-01T06:10:55.465432+0000 2024-04-01T06:11:47.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:11:47.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-04-01T06:11:47.450 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:11:47.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:11:47.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:11:47.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:11:47.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:11:47.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:11:47.541 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:11:47.541 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:11:47.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:11:47.821 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:11:47.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:11:47.821 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:11:47.821 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:11:47.927 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313220 2024-04-01T06:11:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313220 2024-04-01T06:11:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313220' 2024-04-01T06:11:47.983 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:11:47.984 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:11:48.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411333 2024-04-01T06:11:48.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411333 2024-04-01T06:11:48.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313220 1-77309411333' 2024-04-01T06:11:48.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:11:48.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-90194313220 2024-04-01T06:11:48.033 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:11:48.034 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:11:48.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-90194313220 2024-04-01T06:11:48.035 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:11:48.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313220 2024-04-01T06:11:48.036 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 90194313220 2024-04-01T06:11:48.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 90194313220' 2024-04-01T06:11:48.036 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:11:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313219 -lt 90194313220 2024-04-01T06:11:48.338 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:11:49.339 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:11:49.339 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:11:49.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313220 -lt 90194313220 2024-04-01T06:11:49.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:11:49.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411333 2024-04-01T06:11:49.641 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:11:49.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:11:49.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411333 2024-04-01T06:11:49.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:11:49.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411333 2024-04-01T06:11:49.644 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 77309411333 2024-04-01T06:11:49.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411333' 2024-04-01T06:11:49.644 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:11:49.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411333 -lt 77309411333 2024-04-01T06:11:49.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:11:49.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:11:49.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:11:50.282 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:11:50.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T06:11:50.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:11:50.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:11:50.581 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:11:50.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T06:11:50.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:11:50.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:11:50.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-04-01T06:11:51.199 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T06:11:51.199 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-04-01T06:11:44.189282+0000 14'30 22:119 [1,0] 1 [1,0] 1 14'30 2024-04-01T06:11:43.491438+0000 14'30 2024-04-01T06:11:43.491438+0000 0 1 periodic scrub scheduled @ 2024-04-02T06:11:43.491438+0000 20 0 2024-04-01T06:11:51.199 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T06:11:51.199 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T06:11:51.199 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T06:11:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-04-01T06:11:51.210 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:11:51.210 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:11:51.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:11:51.512 INFO:tasks.workunit.client.0.smithi067.stderr:1' 2024-04-01T06:11:51.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:11:51.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:11:51.513 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:11:51.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313222 2024-04-01T06:11:51.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313222 2024-04-01T06:11:51.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313222' 2024-04-01T06:11:51.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:11:51.618 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:11:51.722 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411335 2024-04-01T06:11:51.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411335 2024-04-01T06:11:51.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313222 1-77309411335' 2024-04-01T06:11:51.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:11:51.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-90194313222 2024-04-01T06:11:51.723 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:11:51.724 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:11:51.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-90194313222 2024-04-01T06:11:51.725 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:11:51.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313222 2024-04-01T06:11:51.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 90194313222' 2024-04-01T06:11:51.727 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 90194313222 2024-04-01T06:11:51.727 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:11:52.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313221 -lt 90194313222 2024-04-01T06:11:52.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:11:53.013 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:11:53.014 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:11:53.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313222 -lt 90194313222 2024-04-01T06:11:53.309 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:11:53.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-77309411335 2024-04-01T06:11:53.310 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:11:53.311 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:11:53.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-77309411335 2024-04-01T06:11:53.312 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:11:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411335 2024-04-01T06:11:53.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 77309411335' 2024-04-01T06:11:53.313 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 77309411335 2024-04-01T06:11:53.313 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:11:53.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411335 -lt 77309411335 2024-04-01T06:11:53.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-04-01T06:11:53.595 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-04-01T06:11:53.702 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:11:53.702 INFO:tasks.workunit.client.0.smithi067.stdout: "num_bytes": 210, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects": 30, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_object_clones": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_object_copies": 60, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_missing_on_primary": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_missing": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_degraded": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_misplaced": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_unfound": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_dirty": 30, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_whiteouts": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_read": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_read_kb": 0, 2024-04-01T06:11:53.703 INFO:tasks.workunit.client.0.smithi067.stdout: "num_write": 30, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_write_kb": 30, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_scrub_errors": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_shallow_scrub_errors": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_deep_scrub_errors": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_recovered": 20, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_bytes_recovered": 140, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_keys_recovered": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_omap": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_hit_set_archive": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_bytes_hit_set_archive": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush_kb": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict": 0, 2024-04-01T06:11:53.704 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict_kb": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_promote": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush_mode_high": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush_mode_low": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict_mode_some": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict_mode_full": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_pinned": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_legacy_snapsets": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_large_omap_objects": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_manifest": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_omap_bytes": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_omap_keys": 0, 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_repaired": 20 2024-04-01T06:11:53.705 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:11:53.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-04-01T06:11:53.706 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-04-01T06:11:53.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-04-01T06:11:53.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-04-01T06:11:53.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-04-01T06:11:53.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-04-01T06:11:54.079 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "osd": 1, 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "up_from": 18, 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "seq": 77309411335, 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "num_pgs": 1, 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "num_osds": 1, 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_osds": 1, 2024-04-01T06:11:54.090 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_omap_osds": 1, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "kb": 104857600, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used": 28536, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_data": 304, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_omap": 4, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_meta": 28219, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_avail": 104829064, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "statfs": { 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "total": 107374182400, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "available": 107344961536, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "internally_reserved": 0, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "allocated": 311296, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "data_stored": 49061, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed": 0, 2024-04-01T06:11:54.091 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_allocated": 0, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_original": 0, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "omap_allocated": 4397, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "internal_metadata": 28896979 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "hb_peers": [ 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: 0 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "snap_trim_queue_len": 0, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "num_snap_trimming": 0, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "num_shards_repaired": 10, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "op_queue_age_hist": { 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "histogram": [], 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "upper_bound": 1 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:54.092 INFO:tasks.workunit.client.0.smithi067.stdout: "perf_stat": { 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ms": 0, 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ms": 0, 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ns": 0, 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ns": 0 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout: "alerts": [] 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-04-01T06:11:54.093 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-04-01T06:11:54.360 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:11:54.372 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-04-01T06:11:54.372 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-04-01T06:11:54.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-04-01T06:11:54.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-04-01T06:11:54.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-04-01T06:11:54.639 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "osd": 0, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "up_from": 21, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "seq": 90194313223, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "num_pgs": 1, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "num_osds": 1, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_osds": 1, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_omap_osds": 1, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "kb": 104857600, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used": 28536, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_data": 304, 2024-04-01T06:11:54.650 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_omap": 4, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_meta": 28219, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_avail": 104829064, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "statfs": { 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "total": 107374182400, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "available": 107344961536, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "internally_reserved": 0, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "allocated": 311296, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "data_stored": 49061, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed": 0, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_allocated": 0, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_original": 0, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "omap_allocated": 4401, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "internal_metadata": 28896975 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:54.651 INFO:tasks.workunit.client.0.smithi067.stdout: "hb_peers": [ 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: 1 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "snap_trim_queue_len": 0, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "num_snap_trimming": 0, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "num_shards_repaired": 10, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "op_queue_age_hist": { 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "histogram": [], 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "upper_bound": 1 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "perf_stat": { 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ms": 0, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ms": 0, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ns": 0, 2024-04-01T06:11:54.652 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ns": 0 2024-04-01T06:11:54.653 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:54.653 INFO:tasks.workunit.client.0.smithi067.stdout: "alerts": [] 2024-04-01T06:11:54.653 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:11:54.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-04-01T06:11:54.653 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-04-01T06:11:54.917 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:11:54.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-04-01T06:11:54.929 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-04-01T06:11:54.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-04-01T06:11:54.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-04-01T06:11:54.930 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-04-01T06:11:55.196 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "up_from": 0, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "seq": 0, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "num_pgs": 2, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "num_osds": 2, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_osds": 2, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_omap_osds": 2, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "kb": 209715200, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used": 57072, 2024-04-01T06:11:55.208 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_data": 608, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_omap": 8, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_meta": 56439, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_avail": 209658128, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "statfs": { 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "total": 214748364800, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "available": 214689923072, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "internally_reserved": 0, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "allocated": 622592, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "data_stored": 98122, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed": 0, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_allocated": 0, 2024-04-01T06:11:55.209 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_original": 0, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "omap_allocated": 8798, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "internal_metadata": 57793954 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "hb_peers": [], 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "snap_trim_queue_len": 0, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "num_snap_trimming": 0, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "num_shards_repaired": 20, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "op_queue_age_hist": { 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "histogram": [], 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "upper_bound": 1 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "perf_stat": { 2024-04-01T06:11:55.210 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ms": 0, 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ms": 0, 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ns": 0, 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ns": 0 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout: "alerts": [], 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout: "network_ping_times": [] 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:11:55.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-04-01T06:11:55.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-01T06:11:55.494 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:11:55.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-04-01T06:11:55.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-04-01T06:11:55.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-04-01T06:11:55.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:11:55.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:11:55.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:11:55.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:11:55.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:11:55.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:11:55.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:11:55.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:11:55.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:11:55.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:11:55.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:11:55.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:11:55.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:11:55.617 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:11:55.617 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:11:55.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:11:55.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:11:55.618 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:11:55.619 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:11:55.619 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:11:55.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:11:55.620 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:11:55.633 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:11:55.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:55.634 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:55.634 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:11:55.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:11:55.636 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:11:55.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:11:55.637 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:11:55.637 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:11:55.638 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:11:55.639 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:11:55.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:11:55.640 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:11:55.640 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:11:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:11:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:11:55.641 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:11:55.642 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:11:55.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:11:55.643 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-01T06:11:55.644 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:11:55.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:11:55.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:55.645 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:55.645 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:11:55.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:11:55.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:11:55.646 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-04-01T06:11:55.647 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-01T06:11:55.647 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:55.647 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:55.648 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70255 2024-04-01T06:11:55.650 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-04-01T06:11:55.651 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-04-01T06:11:55.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-01T06:11:55.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-01T06:11:55.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-01T06:11:55.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-04-01T06:11:55.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-04-01T06:11:56.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:56.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-01T06:11:56.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-01T06:11:56.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-01T06:11:56.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:11:56.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:11:56.230 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-01T06:11:56.231 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-01T06:11:56.231 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:11:56.232 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get fsid 2024-04-01T06:11:56.320 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-01T06:11:56.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-01T06:11:56.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-01T06:11:56.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-01T06:11:56.321 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-01T06:11:56.321 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-01T06:11:56.321 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70255/ceph-mon.a.asok 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-01T06:11:56.322 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70255/ceph-mon.a.asok config get mon_host 2024-04-01T06:11:56.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-04-01T06:11:56.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-04-01T06:11:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-01T06:11:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-01T06:11:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-01T06:11:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-04-01T06:11:56.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-01T06:11:57.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-01T06:11:57.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:11:57.224 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:11:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:11:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:57.224 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:57.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:57.225 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:11:57.225 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-01T06:11:57.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:11:57.254 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-04-01T06:11:57.255 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-04-01T06:11:57.263 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T06:11:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:11:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:11:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:11:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-01T06:11:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:11:57.264 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:11:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:11:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:11:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:11:57.265 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:11:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:11:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:11:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:11:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:11:57.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:11:57.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:11:57.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:11:57.267 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:11:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:11:57.267 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:11:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:11:57.268 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:11:57.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:11:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:11:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:11:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:11:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:11:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:11:57.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:11:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:11:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:11:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:11:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:11:57.270 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:11:57.270 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:11:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=54d41253-95e9-469f-a655-7250d531f31e 2024-04-01T06:11:57.271 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 54d41253-95e9-469f-a655-7250d531f31e' 2024-04-01T06:11:57.271 INFO:tasks.workunit.client.0.smithi067.stdout:add osd0 54d41253-95e9-469f-a655-7250d531f31e 2024-04-01T06:11:57.272 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:11:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtUApm9ZqPEBAAMOOK0/nmVzJ/pC9gNbM+Ow== 2024-04-01T06:11:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtUApm9ZqPEBAAMOOK0/nmVzJ/pC9gNbM+Ow=="}' 2024-04-01T06:11:57.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 54d41253-95e9-469f-a655-7250d531f31e -i td/osd-scrub-repair/0/new.json 2024-04-01T06:11:57.487 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:11:57.497 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-04-01T06:11:57.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAtUApm9ZqPEBAAMOOK0/nmVzJ/pC9gNbM+Ow== --osd-uuid 54d41253-95e9-469f-a655-7250d531f31e 2024-04-01T06:11:57.527 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.526+0000 7fe8d950d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:57.538 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.537+0000 7fe8d950d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:57.539 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.539+0000 7fe8d950d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:11:57.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.539+0000 7fe8d950d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:11:57.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.539+0000 7fe8d950d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:11:57.540 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.539+0000 7fe8d950d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-04-01T06:11:57.630 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:11:57.629+0000 7fe8d950d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-04-01T06:12:03.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-04-01T06:12:03.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:12:03.280 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd0 key to auth repository 2024-04-01T06:12:03.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-01T06:12:03.281 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:12:03.651 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:12:03.652 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-01T06:12:03.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:12:03.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:12:03.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:12:03.653 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:12:03.679 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:03.679+0000 7ff024af7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:03.681 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:03.681+0000 7ff024af7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:03.682 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:03.682+0000 7ff024af7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:03.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:12:03.938 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:12:03.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:03.938 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:04.200 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:04.200+0000 7ff024af7640 -1 Falling back to public interface 2024-04-01T06:12:04.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:05.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:05.218 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:12:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:12:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:05.219 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:05.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:06.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:06.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:06.511 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:12:06.512 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:12:06.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:06.512 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:06.529 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:06.529+0000 7ff024af7640 -1 osd.0 0 log_to_monitors true 2024-04-01T06:12:06.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:07.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:07.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:07.957 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:12:07.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:12:07.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:07.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:08.278 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:09.279 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:09.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:09.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:12:09.280 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:12:09.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:09.280 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:09.574 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:10.578 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:12:10.579 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:12:10.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:10.579 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:10.868 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:11.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:11.870 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:11.870 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:12:11.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:12:11.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:11.871 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:12.253 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:13.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:13.254 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:13.254 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:12:13.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:12:13.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:13.255 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:13.549 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:14.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:14.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:14.551 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:12:14.551 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:12:14.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:14.552 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:14.843 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:15.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:15.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:15.845 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:12:15.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:12:15.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:15.845 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:16.136 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:17.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:17.138 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:17.138 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:12:17.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:12:17.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:17.139 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:17.523 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:18.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:18.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:18.525 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:12:18.525 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:12:18.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:18.526 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:19.020 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:20.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:20.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:20.021 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:12:20.022 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:12:20.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:20.022 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:20.697 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:21.699 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:21.699 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:12:21.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:12:21.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:21.700 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:22.274 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:23.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:23.276 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:23.276 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:12:23.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:12:23.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:23.277 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:23.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:24.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:24.986 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:24.986 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:12:24.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:12:24.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:24.987 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:25.580 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:26.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:26.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:26.581 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:12:26.582 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:12:26.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:26.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:27.315 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:28.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:28.316 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:28.316 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:12:28.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:12:28.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:28.317 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:12:28.711 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1173085997,v1:127.0.0.1:6803/1173085997] [v2:127.0.0.1:6804/1173085997,v1:127.0.0.1:6805/1173085997] exists,up 54d41253-95e9-469f-a655-7250d531f31e 2024-04-01T06:12:28.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:12:28.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:12:28.715 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:12:28.716 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:12:28.717 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52d4a775-0fe1-43d4-a73d-d2feae5abd94 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 52d4a775-0fe1-43d4-a73d-d2feae5abd94' 2024-04-01T06:12:28.718 INFO:tasks.workunit.client.0.smithi067.stdout:add osd1 52d4a775-0fe1-43d4-a73d-d2feae5abd94 2024-04-01T06:12:28.719 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:12:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMUApmf4OyKxAAu8DI2GGXB8W3XoGgfHzPZQ== 2024-04-01T06:12:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMUApmf4OyKxAAu8DI2GGXB8W3XoGgfHzPZQ=="}' 2024-04-01T06:12:28.733 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52d4a775-0fe1-43d4-a73d-d2feae5abd94 -i td/osd-scrub-repair/1/new.json 2024-04-01T06:12:29.051 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:12:29.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-04-01T06:12:29.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBMUApmf4OyKxAAu8DI2GGXB8W3XoGgfHzPZQ== --osd-uuid 52d4a775-0fe1-43d4-a73d-d2feae5abd94 2024-04-01T06:12:29.093 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.093+0000 7f409ad06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:29.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.144+0000 7f409ad06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:29.145 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.146+0000 7f409ad06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:29.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.146+0000 7f409ad06640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:12:29.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.146+0000 7f409ad06640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:12:29.146 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.146+0000 7f409ad06640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-04-01T06:12:29.219 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:29.220+0000 7f409ad06640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-04-01T06:12:35.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-04-01T06:12:35.035 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:12:35.036 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd1 key to auth repository 2024-04-01T06:12:35.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-01T06:12:35.036 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:12:35.406 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-01T06:12:35.406 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:12:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:12:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:12:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:12:35.407 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:12:35.434 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:35.432+0000 7f4a53413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:35.435 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:35.435+0000 7f4a53413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:35.436 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:35.436+0000 7f4a53413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:12:35.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-01T06:12:35.690 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:12:35.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:12:35.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:12:35.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:12:35.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:35.691 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:12:35.691 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:12:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:35.692 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:35.961 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:35.960+0000 7f4a53413640 -1 Falling back to public interface 2024-04-01T06:12:35.974 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:36.975 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:36.976 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:12:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:12:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:36.976 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:37.266 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:38.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:38.268 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:38.268 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:12:38.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:12:38.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:38.269 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:38.293 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:12:38.292+0000 7f4a53413640 -1 osd.1 0 log_to_monitors true 2024-04-01T06:12:38.562 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:39.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:39.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:39.564 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:12:39.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:12:39.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:39.565 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:39.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:40.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:40.909 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:12:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:12:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:40.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:41.203 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:42.204 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:42.204 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:12:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:12:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:42.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:42.586 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:43.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:43.587 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:43.587 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:12:43.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:12:43.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:43.588 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:43.924 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:44.925 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:44.926 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:12:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:12:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:44.926 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:45.217 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:46.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:46.218 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:46.219 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:12:46.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:12:46.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:46.220 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:46.534 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:47.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:47.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:47.536 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:12:47.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:12:47.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:47.537 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:47.831 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:48.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:48.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:48.833 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:12:48.834 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:12:48.834 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:48.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:49.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:50.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:50.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:50.128 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:12:50.128 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:12:50.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:50.129 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:50.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:51.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:51.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:51.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:12:51.509 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:12:51.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:51.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:51.933 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:52.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:52.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:52.935 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:12:52.935 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:12:52.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:52.936 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:53.621 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:54.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:54.622 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:54.623 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:12:54.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:12:54.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:54.623 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:54.982 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:55.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:55.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:55.984 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:12:55.984 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:12:55.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:55.985 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:56.758 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:57.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:57.759 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:57.760 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:12:57.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:12:57.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:57.760 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:58.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:12:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:12:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:12:59.369 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:12:59.369 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:12:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:12:59.370 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:12:59.907 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:00.908 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:00.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:00.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:13:00.909 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:13:00.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:00.909 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:13:01.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:02.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:02.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:02.209 INFO:tasks.workunit.client.0.smithi067.stdout:19 2024-04-01T06:13:02.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-01T06:13:02.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:02.209 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:13:02.499 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2057623442,v1:127.0.0.1:6811/2057623442] [v2:127.0.0.1:6812/2057623442,v1:127.0.0.1:6813/2057623442] exists,up 52d4a775-0fe1-43d4-a73d-d2feae5abd94 2024-04-01T06:13:02.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:13:02.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:13:02.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:13:02.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T06:13:02.505 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:13:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:13:02.507 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2637da6e-0d72-4bca-a7e8-0a17cf59f6ce 2024-04-01T06:13:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2637da6e-0d72-4bca-a7e8-0a17cf59f6ce' 2024-04-01T06:13:02.509 INFO:tasks.workunit.client.0.smithi067.stdout:add osd2 2637da6e-0d72-4bca-a7e8-0a17cf59f6ce 2024-04-01T06:13:02.509 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-01T06:13:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuUApmLvb/HhAABAIZOFavqV0BTnMhB5V3LA== 2024-04-01T06:13:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuUApmLvb/HhAABAIZOFavqV0BTnMhB5V3LA=="}' 2024-04-01T06:13:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2637da6e-0d72-4bca-a7e8-0a17cf59f6ce -i td/osd-scrub-repair/2/new.json 2024-04-01T06:13:02.894 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:13:02.905 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-04-01T06:13:02.906 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBuUApmLvb/HhAABAIZOFavqV0BTnMhB5V3LA== --osd-uuid 2637da6e-0d72-4bca-a7e8-0a17cf59f6ce 2024-04-01T06:13:02.935 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:02.935+0000 7f1642cd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:13:02.970 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:02.970+0000 7f1642cd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:13:02.971 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:02.971+0000 7f1642cd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:13:02.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:02.972+0000 7f1642cd3640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:13:02.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:02.972+0000 7f1642cd3640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:13:02.972 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:02.972+0000 7f1642cd3640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-04-01T06:13:03.042 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:03.042+0000 7f1642cd3640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-04-01T06:13:09.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-04-01T06:13:09.119 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-01T06:13:09.120 INFO:tasks.workunit.client.0.smithi067.stdout:adding osd2 key to auth repository 2024-04-01T06:13:09.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-01T06:13:09.121 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-01T06:13:09.491 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.2 2024-04-01T06:13:09.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-01T06:13:09.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:13:09.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-01T06:13:09.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-01T06:13:09.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-01T06:13:09.519 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:09.519+0000 7f2f6d793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:13:09.521 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:09.521+0000 7f2f6d793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:13:09.522 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:09.522+0000 7f2f6d793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:13:09.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:13:09.779 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:13:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:09.780 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:10.037 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:10.036+0000 7f2f6d793640 -1 Falling back to public interface 2024-04-01T06:13:10.062 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:11.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:11.063 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:11.064 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:13:11.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:13:11.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:11.064 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:11.353 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:12.354 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:12.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:12.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:13:12.355 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:13:12.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:12.355 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:12.487 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:13:12.487+0000 7f2f6d793640 -1 osd.2 0 log_to_monitors true 2024-04-01T06:13:12.723 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:13.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:13.725 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:13.726 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:13:13.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:13:13.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:13.726 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:14.017 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:15.018 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:15.051 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:15.051 INFO:tasks.workunit.client.0.smithi067.stdout:4 2024-04-01T06:13:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-01T06:13:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:15.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:15.325 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:16.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:16.326 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:16.326 INFO:tasks.workunit.client.0.smithi067.stdout:5 2024-04-01T06:13:16.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-01T06:13:16.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:16.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:16.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:17.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:17.661 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:17.662 INFO:tasks.workunit.client.0.smithi067.stdout:6 2024-04-01T06:13:17.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-01T06:13:17.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:17.662 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:17.955 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:18.956 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:18.957 INFO:tasks.workunit.client.0.smithi067.stdout:7 2024-04-01T06:13:18.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:18.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-01T06:13:18.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:18.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:19.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:20.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:20.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:20.401 INFO:tasks.workunit.client.0.smithi067.stdout:8 2024-04-01T06:13:20.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-01T06:13:20.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:20.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:20.702 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:21.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:21.703 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:21.704 INFO:tasks.workunit.client.0.smithi067.stdout:9 2024-04-01T06:13:21.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-01T06:13:21.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:21.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:21.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:22.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:23.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:23.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-01T06:13:23.000 INFO:tasks.workunit.client.0.smithi067.stdout:10 2024-04-01T06:13:23.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:23.000 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:23.327 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:24.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:24.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:24.328 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-01T06:13:24.329 INFO:tasks.workunit.client.0.smithi067.stdout:11 2024-04-01T06:13:24.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:24.329 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:24.813 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:25.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:25.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:25.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-01T06:13:25.815 INFO:tasks.workunit.client.0.smithi067.stdout:12 2024-04-01T06:13:25.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:25.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:26.504 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:27.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:27.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:27.506 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-01T06:13:27.506 INFO:tasks.workunit.client.0.smithi067.stdout:13 2024-04-01T06:13:27.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:27.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:28.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:29.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:29.435 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:29.436 INFO:tasks.workunit.client.0.smithi067.stdout:14 2024-04-01T06:13:29.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-01T06:13:29.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:29.436 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:29.748 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:30.749 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:30.749 INFO:tasks.workunit.client.0.smithi067.stdout:15 2024-04-01T06:13:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-01T06:13:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:30.750 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:31.515 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:32.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:32.516 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:32.516 INFO:tasks.workunit.client.0.smithi067.stdout:16 2024-04-01T06:13:32.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-01T06:13:32.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:32.517 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:33.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:34.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:34.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:34.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-01T06:13:34.170 INFO:tasks.workunit.client.0.smithi067.stdout:17 2024-04-01T06:13:34.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:34.170 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:34.854 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:13:35.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:13:35.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:13:35.855 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-01T06:13:35.856 INFO:tasks.workunit.client.0.smithi067.stdout:18 2024-04-01T06:13:35.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:13:35.856 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-01T06:13:36.148 INFO:tasks.workunit.client.0.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2653959051,v1:127.0.0.1:6819/2653959051] [v2:127.0.0.1:6820/2653959051,v1:127.0.0.1:6821/2653959051] exists,up 2637da6e-0d72-4bca-a7e8-0a17cf59f6ce 2024-04-01T06:13:36.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:13:36.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:13:36.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:13:36.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-04-01T06:13:36.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-04-01T06:13:36.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-04-01T06:13:36.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-04-01T06:13:36.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-04-01T06:13:36.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-01T06:13:36.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-04-01T06:13:36.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-04-01T06:13:37.353 INFO:tasks.workunit.client.0.smithi067.stderr:pool 'testpool' created 2024-04-01T06:13:37.365 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-01T06:13:38.367 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-04-01T06:13:38.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-04-01T06:13:38.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:13:38.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:13:38.368 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:13:38.368 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:13:38.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:13:38.369 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:13:38.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:13:38.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:13:38.369 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:13:38.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:13:38.470 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:13:38.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:13:38.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:13:38.471 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:13:38.471 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:13:38.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:13:38.755 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:13:38.755 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:13:38.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:13:38.756 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:13:38.756 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:13:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-04-01T06:13:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-04-01T06:13:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-04-01T06:13:38.937 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:13:38.937 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:13:39.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-04-01T06:13:39.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-04-01T06:13:39.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672971' 2024-04-01T06:13:39.043 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:13:39.043 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:13:39.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-04-01T06:13:39.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-04-01T06:13:39.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672971 2-64424509447' 2024-04-01T06:13:39.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:13:39.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-04-01T06:13:39.166 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:13:39.166 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:13:39.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-04-01T06:13:39.167 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:13:39.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-04-01T06:13:39.169 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 21474836502 2024-04-01T06:13:39.169 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-01T06:13:39.169 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:13:39.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-04-01T06:13:39.453 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:13:40.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:13:40.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:13:40.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836502 2024-04-01T06:13:40.751 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:13:40.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-04-01T06:13:40.752 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:13:40.753 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:13:40.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-04-01T06:13:40.754 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:13:40.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-04-01T06:13:40.755 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-04-01T06:13:40.755 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 42949672971 2024-04-01T06:13:40.755 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:13:41.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-04-01T06:13:41.037 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:13:41.037 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-04-01T06:13:41.038 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:13:41.038 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:13:41.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-04-01T06:13:41.039 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:13:41.040 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-04-01T06:13:41.041 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509447 2024-04-01T06:13:41.041 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-04-01T06:13:41.041 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:13:41.323 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-04-01T06:13:41.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:13:41.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:13:41.324 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:13:41.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T06:13:41.683 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:13:41.683 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:13:41.683 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:13:41.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:13:41.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:13:41.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:13:41.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:13:41.962 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T06:13:41.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:13:41.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:13:41.963 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:13:42.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T06:13:42.313 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:42.314 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:42.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:42.461 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:42.704 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:42.705 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:42.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:42.931 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:43.149 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:43.150 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:43.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:43.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:43.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:43.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:43.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:43.762 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:43.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:43.972 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:44.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:44.173 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:44.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:44.408 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:44.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:44.635 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:44.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:44.811 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:45.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:45.012 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:45.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:45.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:45.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:45.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:45.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:45.642 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:45.835 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:46.052 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:46.053 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:46.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:46.229 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:46.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:46.480 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:46.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:46.707 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:46.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:46.900 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:47.126 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:47.127 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:47.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:47.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:47.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:47.529 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:47.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:47.764 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:47.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:47.973 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:48.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:48.200 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:48.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-04-01T06:13:48.418 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-04-01T06:13:48.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-04-01T06:13:48.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-04-01T06:13:48.629 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-01T06:13:48.629 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-04-01T06:13:48.629 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T06:13:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T06:13:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T06:13:48.630 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T06:13:48.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-01T06:13:48.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-04-01T06:13:48.923 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-01T06:13:49.211 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-04-01T06:13:49.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-04-01T06:13:49.211 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-04-01T06:13:49.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-01T06:13:49.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-04-01T06:13:49.212 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-01T06:13:49.496 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-04-01T06:13:49.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-04-01T06:13:49.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-04-01T06:13:49.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-01T06:13:49.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-04-01T06:13:49.497 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-01T06:13:49.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-04-01T06:13:49.782 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-04-01T06:13:49.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:13:49.783 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:13:49.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:13:49.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:13:49.783 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:13:50.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:13:50.090 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-04-01T06:13:50.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:13:50.091 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:13:50.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:13:50.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:13:50.091 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:13:50.397 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:13:50.398 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-04-01T06:13:50.398 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:50.399 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-04-01T06:13:50.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:13:50.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-04-01T06:13:50.400 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:50.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:50.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:13:50.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:50.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:13:50.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-04-01T06:13:51.790 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-04-01T06:13:52.304 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:52.304 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-04-01T06:13:52.306 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:13:52.307 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-04-01T06:13:53.339 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:104778fc:::obj2:head# 2024-04-01T06:13:53.858 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:53.858 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-04-01T06:13:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:13:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-04-01T06:13:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:13:53.860 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:53.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:13:53.861 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-04-01T06:13:54.689 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-04-01T06:13:55.205 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:55.205 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-04-01T06:13:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:13:55.207 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-04-01T06:13:55.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:55.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:55.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:13:55.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:55.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:13:55.208 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-04-01T06:13:55.974 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-04-01T06:13:56.492 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:56.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-04-01T06:13:56.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:13:56.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-04-01T06:13:56.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:56.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:56.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:13:56.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:56.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:13:56.495 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-04-01T06:13:57.151 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:c52c9666:::obj5:head# 2024-04-01T06:13:57.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:57.666 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-04-01T06:13:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:13:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-04-01T06:13:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:57.668 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:13:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:13:57.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-04-01T06:13:58.329 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:233a42c1:::obj6:head# 2024-04-01T06:13:58.844 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:58.844 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-04-01T06:13:58.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:13:58.846 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-04-01T06:13:58.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:13:58.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:13:58.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:13:58.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:13:58.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:13:58.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-04-01T06:13:59.480 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:9a09113a:::obj7:head# 2024-04-01T06:13:59.999 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:13:59.999 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-04-01T06:14:00.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:00.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-04-01T06:14:00.001 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:00.002 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-04-01T06:14:00.733 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:a557efb1:::obj8:head# 2024-04-01T06:14:01.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:01.248 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-04-01T06:14:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-04-01T06:14:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:01.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:01.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:01.251 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-04-01T06:14:01.894 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:fe351e27:::obj9:head# 2024-04-01T06:14:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:02.409 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-04-01T06:14:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-04-01T06:14:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-04-01T06:14:03.046 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:d1337354:::obj10:head# 2024-04-01T06:14:03.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:03.562 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-04-01T06:14:03.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:03.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-04-01T06:14:04.165 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-04-01T06:14:04.929 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:04.930 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:04.932 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-04-01T06:14:05.602 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-04-01T06:14:06.371 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:06.372 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-04-01T06:14:06.373 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:06.374 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-04-01T06:14:06.988 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-04-01T06:14:07.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:07.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-04-01T06:14:07.509 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:07.510 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-04-01T06:14:08.115 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:e0a44829:::obj14:head# 2024-04-01T06:14:08.630 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:08.630 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-04-01T06:14:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:08.632 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-04-01T06:14:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:08.633 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-04-01T06:14:09.225 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:ab946124:::obj15:head# 2024-04-01T06:14:09.743 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:09.743 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-04-01T06:14:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-04-01T06:14:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:09.745 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:09.746 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-04-01T06:14:10.384 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-04-01T06:14:10.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:10.902 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-04-01T06:14:10.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:10.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-04-01T06:14:10.902 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:10.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:10.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:10.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:10.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:10.903 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-04-01T06:14:11.579 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-04-01T06:14:12.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:12.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:12.096 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:12.097 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-04-01T06:14:12.773 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:b9836a99:::obj18:head# 2024-04-01T06:14:13.288 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:13.288 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-04-01T06:14:13.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:13.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-04-01T06:14:13.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:13.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:13.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:13.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:13.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:13.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-04-01T06:14:13.916 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-04-01T06:14:14.431 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:14.431 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-04-01T06:14:14.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:14.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-04-01T06:14:14.433 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:14.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:14.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:14.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:14.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:14.434 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-04-01T06:14:15.043 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:24b5611b:::obj20:head# 2024-04-01T06:14:15.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:15.559 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:15.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-04-01T06:14:16.220 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-04-01T06:14:16.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:16.739 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-04-01T06:14:16.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:16.741 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-04-01T06:14:17.398 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:c0c49def:::obj22:head# 2024-04-01T06:14:17.912 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:17.912 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-04-01T06:14:17.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:17.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-04-01T06:14:17.914 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:17.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:17.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:17.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:17.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:17.915 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-04-01T06:14:18.650 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:242c40e0:::obj23:head# 2024-04-01T06:14:19.165 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:19.165 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-04-01T06:14:19.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:19.167 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-04-01T06:14:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:19.168 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-04-01T06:14:19.860 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:7eacede5:::obj24:head# 2024-04-01T06:14:20.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:20.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:20.378 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-04-01T06:14:21.046 INFO:tasks.workunit.client.0.smithi067.stdout:remove 0#1:7109478f:::obj25:head# 2024-04-01T06:14:21.561 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-04-01T06:14:21.561 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-04-01T06:14:21.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-04-01T06:14:21.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-04-01T06:14:21.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-04-01T06:14:21.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-01T06:14:21.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-01T06:14:21.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-01T06:14:21.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:21.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-04-01T06:14:22.215 INFO:tasks.workunit.client.0.smithi067.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-04-01T06:14:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:14:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:14:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:14:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-01T06:14:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:14:22.736 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:14:22.737 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-04-01T06:14:22.738 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-01T06:14:22.739 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.1 2024-04-01T06:14:22.739 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:14:22.740 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-04-01T06:14:22.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-01T06:14:22.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:14:22.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:14:22.740 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:14:22.768 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:22.767+0000 7fce238aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:14:22.770 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:22.770+0000 7fce238aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:14:22.774 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:22.771+0000 7fce238aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:14:23.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-01T06:14:23.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:14:23.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-01T06:14:23.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:14:23.032 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:14:23.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:23.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:14:23.033 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:14:23.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:23.033 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:14:23.287 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:23.287+0000 7fce238aa640 -1 Falling back to public interface 2024-04-01T06:14:23.318 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:14:24.319 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:14:24.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:24.320 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:14:24.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:14:24.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:24.320 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:14:24.613 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:14:25.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:14:25.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:25.614 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:14:25.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:14:25.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:25.615 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:14:25.694 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:25.694+0000 7fce238aa640 -1 osd.1 21 log_to_monitors true 2024-04-01T06:14:25.957 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:14:26.958 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:14:26.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:26.959 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:14:26.959 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:14:26.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:26.960 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-01T06:14:27.243 INFO:tasks.workunit.client.0.smithi067.stdout:osd.1 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [10,21) [v2:127.0.0.1:6802/2041935437,v1:127.0.0.1:6803/2041935437] [v2:127.0.0.1:6804/2041935437,v1:127.0.0.1:6805/2041935437] exists,up 52d4a775-0fe1-43d4-a73d-d2feae5abd94 2024-04-01T06:14:27.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:14:27.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:14:27.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:14:27.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:14:27.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-01T06:14:27.247 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-01T06:14:27.248 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-01T06:14:27.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-04-01T06:14:27.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-04-01T06:14:27.249 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-01T06:14:27.249 INFO:tasks.workunit.client.0.smithi067.stdout:start osd.0 2024-04-01T06:14:27.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=286f4ea1-530c-4f6c-a10f-569fcc2a507d --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70255/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-04-01T06:14:27.250 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-04-01T06:14:27.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-01T06:14:27.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-01T06:14:27.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-01T06:14:27.250 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-01T06:14:27.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:27.273+0000 7f130bb1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:14:27.274 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:27.275+0000 7f130bb1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:14:27.276 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:27.276+0000 7f130bb1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:27.535 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-01T06:14:27.536 INFO:tasks.workunit.client.0.smithi067.stdout:0 2024-04-01T06:14:27.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:27.536 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:14:27.797 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:27.796+0000 7f130bb1d640 -1 Falling back to public interface 2024-04-01T06:14:27.815 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:14:28.816 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:14:28.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:28.817 INFO:tasks.workunit.client.0.smithi067.stdout:1 2024-04-01T06:14:28.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-01T06:14:28.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:28.817 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:14:29.109 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:14:30.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:14:30.111 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:30.111 INFO:tasks.workunit.client.0.smithi067.stdout:2 2024-04-01T06:14:30.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-01T06:14:30.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:30.112 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:14:30.140 INFO:tasks.workunit.client.0.smithi067.stderr:2024-04-01T06:14:30.140+0000 7f130bb1d640 -1 osd.0 20 log_to_monitors true 2024-04-01T06:14:30.456 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-01T06:14:31.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-01T06:14:31.458 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-01T06:14:31.458 INFO:tasks.workunit.client.0.smithi067.stdout:3 2024-04-01T06:14:31.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-01T06:14:31.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-01T06:14:31.459 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-01T06:14:31.771 INFO:tasks.workunit.client.0.smithi067.stdout:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6810/607508900,v1:127.0.0.1:6811/607508900] [v2:127.0.0.1:6812/607508900,v1:127.0.0.1:6813/607508900] exists,up 54d41253-95e9-469f-a655-7250d531f31e 2024-04-01T06:14:31.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-01T06:14:31.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-01T06:14:31.772 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:14:31.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:14:31.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:14:31.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:14:31.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:14:31.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:14:31.863 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:14:31.863 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:14:32.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:14:32.237 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:14:32.237 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:14:32.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:14:32.237 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:32.237 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:14:32.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084290 2024-04-01T06:14:32.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084290 2024-04-01T06:14:32.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290' 2024-04-01T06:14:32.389 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:32.389 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:14:32.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182403 2024-04-01T06:14:32.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182403 2024-04-01T06:14:32.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-107374182403' 2024-04-01T06:14:32.498 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:32.498 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:14:32.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-04-01T06:14:32.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-04-01T06:14:32.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-107374182403 2-64424509459' 2024-04-01T06:14:32.603 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:32.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084290 2024-04-01T06:14:32.604 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:32.604 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:14:32.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084290 2024-04-01T06:14:32.605 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:32.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084290 2024-04-01T06:14:32.606 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2024-04-01T06:14:32.607 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 120259084290 2024-04-01T06:14:32.607 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:14:32.888 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 120259084290 2024-04-01T06:14:32.889 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:14:33.890 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:14:33.890 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:14:34.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084290 -lt 120259084290 2024-04-01T06:14:34.189 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:34.190 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-107374182403 2024-04-01T06:14:34.190 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:34.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:14:34.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-107374182403 2024-04-01T06:14:34.191 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:34.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182403 2024-04-01T06:14:34.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 107374182403' 2024-04-01T06:14:34.193 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 107374182403 2024-04-01T06:14:34.193 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:14:34.484 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182403 -lt 107374182403 2024-04-01T06:14:34.485 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:34.485 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-04-01T06:14:34.485 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:34.486 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:14:34.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-04-01T06:14:34.487 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-04-01T06:14:34.488 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-04-01T06:14:34.488 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509459 2024-04-01T06:14:34.489 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:14:34.769 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-04-01T06:14:34.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:14:34.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:14:34.770 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:14:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T06:14:35.122 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:14:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:14:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:14:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:14:35.122 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:14:35.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:14:35.123 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:14:35.404 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T06:14:35.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:14:35.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:14:35.404 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-04-01T06:14:35.773 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:14:35.774 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:14:35.774 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:14:35.774 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:14:36.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:36.055 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-04-01T06:14:36.322 INFO:tasks.workunit.client.0.smithi067.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-04-01T06:14:36.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:36.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-04-01T06:14:36.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:36.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-04-01T06:14:36.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-04-01T06:14:36.334 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:14:36.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:14:36.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:14:36.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:14:36.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:14:36.335 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:14:36.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:13:37.176820+0000 '>' 2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:36.659 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:14:37.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:14:37.660 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:14:37.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:14:37.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:14:37.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:14:37.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:14:37.661 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:14:37.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:13:37.176820+0000 '>' 2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:37.995 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:14:38.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:14:38.996 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:14:38.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:14:38.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:14:38.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:14:38.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:14:38.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:14:39.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:13:37.176820+0000 '>' 2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:39.289 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:14:40.290 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:14:40.291 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:14:40.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:14:40.291 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:14:40.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:14:40.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:14:40.292 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:14:40.582 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:13:37.176820+0000 '>' 2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:40.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-04-01T06:14:41.583 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-04-01T06:14:41.584 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-04-01T06:14:41.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-04-01T06:14:41.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-04-01T06:14:41.584 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-04-01T06:14:41.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-04-01T06:14:41.585 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-04-01T06:14:36.811578+0000 '>' 2024-04-01T06:13:37.176820+0000 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-01T06:14:41.874 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-01T06:14:41.875 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-01T06:14:41.964 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-01T06:14:42.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-01T06:14:42.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-01T06:14:42.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-01T06:14:42.006 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:14:42.006 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:14:42.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:14:42.246 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:14:42.246 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:14:42.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:14:42.246 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:42.246 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:14:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084293 2024-04-01T06:14:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084293 2024-04-01T06:14:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293' 2024-04-01T06:14:42.351 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:42.351 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:14:42.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-04-01T06:14:42.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-04-01T06:14:42.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293 1-107374182406' 2024-04-01T06:14:42.457 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:42.458 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:14:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-04-01T06:14:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-04-01T06:14:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293 1-107374182406 2-64424509462' 2024-04-01T06:14:42.563 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:42.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084293 2024-04-01T06:14:42.563 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:42.564 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:14:42.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084293 2024-04-01T06:14:42.565 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084293 2024-04-01T06:14:42.566 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084293' 2024-04-01T06:14:42.566 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 120259084293 2024-04-01T06:14:42.567 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:14:42.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084292 -lt 120259084293 2024-04-01T06:14:42.847 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:14:43.848 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:14:43.848 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:14:44.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084293 -lt 120259084293 2024-04-01T06:14:44.145 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:44.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-107374182406 2024-04-01T06:14:44.145 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:44.146 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:14:44.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-107374182406 2024-04-01T06:14:44.147 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:44.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-04-01T06:14:44.148 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 107374182406' 2024-04-01T06:14:44.149 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 107374182406 2024-04-01T06:14:44.149 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:14:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-04-01T06:14:44.451 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:44.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-04-01T06:14:44.451 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:44.452 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:14:44.453 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-04-01T06:14:44.453 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:44.454 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-04-01T06:14:44.454 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509462 2024-04-01T06:14:44.455 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-04-01T06:14:44.455 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:14:44.737 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-04-01T06:14:44.737 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-01T06:14:44.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:14:44.738 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-01T06:14:45.094 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-01T06:14:45.375 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-01T06:14:45.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-01T06:14:45.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-01T06:14:45.376 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-01T06:14:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-01T06:14:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-01T06:14:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-01T06:14:45.735 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-04-01T06:14:45.999 INFO:tasks.workunit.client.0.smithi067.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-01T06:14:46.000 INFO:tasks.workunit.client.0.smithi067.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-04-01T06:14:38.016793+0000 20'30 29:130 [1,0,2] 1 [1,0,2] 1 20'30 2024-04-01T06:14:36.811578+0000 20'30 2024-04-01T06:14:36.811578+0000 0 1 periodic scrub scheduled @ 2024-04-02T06:14:36.811578+0000 17 0 2024-04-01T06:14:46.000 INFO:tasks.workunit.client.0.smithi067.stdout: 2024-04-01T06:14:46.000 INFO:tasks.workunit.client.0.smithi067.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-01T06:14:46.000 INFO:tasks.workunit.client.0.smithi067.stderr:dumped pgs 2024-04-01T06:14:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-04-01T06:14:46.011 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-04-01T06:14:46.011 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-04-01T06:14:46.296 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-04-01T06:14:46.296 INFO:tasks.workunit.client.0.smithi067.stderr:1 2024-04-01T06:14:46.296 INFO:tasks.workunit.client.0.smithi067.stderr:2' 2024-04-01T06:14:46.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-04-01T06:14:46.297 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:46.297 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-01T06:14:46.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084295 2024-04-01T06:14:46.401 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084295 2024-04-01T06:14:46.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295' 2024-04-01T06:14:46.402 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:46.402 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-01T06:14:46.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182408 2024-04-01T06:14:46.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182408 2024-04-01T06:14:46.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295 1-107374182408' 2024-04-01T06:14:46.507 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-04-01T06:14:46.508 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-01T06:14:46.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-04-01T06:14:46.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-04-01T06:14:46.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295 1-107374182408 2-64424509464' 2024-04-01T06:14:46.612 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:46.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084295 2024-04-01T06:14:46.613 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:46.614 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-04-01T06:14:46.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084295 2024-04-01T06:14:46.614 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:46.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084295 2024-04-01T06:14:46.616 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-04-01T06:14:46.616 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.0 seq 120259084295 2024-04-01T06:14:46.616 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:14:46.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084293 -lt 120259084295 2024-04-01T06:14:46.898 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-04-01T06:14:47.899 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-01T06:14:47.899 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-01T06:14:48.191 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084295 -lt 120259084295 2024-04-01T06:14:48.192 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:48.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-107374182408 2024-04-01T06:14:48.192 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:48.193 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-04-01T06:14:48.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-107374182408 2024-04-01T06:14:48.194 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:48.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182408 2024-04-01T06:14:48.195 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 107374182408' 2024-04-01T06:14:48.195 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.1 seq 107374182408 2024-04-01T06:14:48.196 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-01T06:14:48.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182408 -lt 107374182408 2024-04-01T06:14:48.490 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-04-01T06:14:48.490 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-04-01T06:14:48.490 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-04-01T06:14:48.491 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-04-01T06:14:48.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-04-01T06:14:48.492 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-04-01T06:14:48.493 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-04-01T06:14:48.493 INFO:tasks.workunit.client.0.smithi067.stdout:waiting osd.2 seq 64424509464 2024-04-01T06:14:48.494 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-04-01T06:14:48.494 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-01T06:14:48.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-04-01T06:14:48.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-04-01T06:14:48.778 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_bytes": 210, 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects": 30, 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_object_clones": 0, 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_object_copies": 90, 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_missing_on_primary": 0, 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_missing": 0, 2024-04-01T06:14:48.885 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_degraded": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_misplaced": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_unfound": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_dirty": 30, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_whiteouts": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_read": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_read_kb": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_write": 30, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_write_kb": 30, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_scrub_errors": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_shallow_scrub_errors": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_deep_scrub_errors": 0, 2024-04-01T06:14:48.886 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_recovered": 26, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_bytes_recovered": 182, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_keys_recovered": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_omap": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_hit_set_archive": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_bytes_hit_set_archive": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush_kb": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict_kb": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_promote": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush_mode_high": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_flush_mode_low": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict_mode_some": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_evict_mode_full": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_pinned": 0, 2024-04-01T06:14:48.887 INFO:tasks.workunit.client.0.smithi067.stdout: "num_legacy_snapsets": 0, 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stdout: "num_large_omap_objects": 0, 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_manifest": 0, 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stdout: "num_omap_bytes": 0, 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stdout: "num_omap_keys": 0, 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stdout: "num_objects_repaired": 26 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-04-01T06:14:48.888 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-04-01T06:14:48.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-04-01T06:14:48.994 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-04-01T06:14:48.995 INFO:tasks.workunit.client.0.smithi067.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-04-01T06:14:48.996 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-04-01T06:14:48.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-01T06:14:48.997 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-04-01T06:14:48.997 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-04-01T06:14:48.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-04-01T06:14:48.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-04-01T06:14:48.998 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-04-01T06:14:49.265 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout:{ 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "osd": 0, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "up_from": 28, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "seq": 120259084295, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "num_pgs": 1, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "num_osds": 1, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_osds": 1, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "num_per_pool_omap_osds": 1, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "kb": 104857600, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used": 28464, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_data": 360, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_omap": 4, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_used_meta": 28091, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "kb_avail": 104829136, 2024-04-01T06:14:49.276 INFO:tasks.workunit.client.0.smithi067.stdout: "statfs": { 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "total": 107374182400, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "available": 107345035264, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "internally_reserved": 0, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "allocated": 368640, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "data_stored": 200601, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed": 0, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_allocated": 0, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "data_compressed_original": 0, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "omap_allocated": 4402, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "internal_metadata": 28765902 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: "hb_peers": [ 2024-04-01T06:14:49.277 INFO:tasks.workunit.client.0.smithi067.stdout: 1, 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: 2 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: ], 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "snap_trim_queue_len": 0, 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "num_snap_trimming": 0, 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "num_shards_repaired": 26, 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "op_queue_age_hist": { 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "histogram": [], 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "upper_bound": 1 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "perf_stat": { 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ms": 0, 2024-04-01T06:14:49.278 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ms": 0, 2024-04-01T06:14:49.279 INFO:tasks.workunit.client.0.smithi067.stdout: "commit_latency_ns": 0, 2024-04-01T06:14:49.279 INFO:tasks.workunit.client.0.smithi067.stdout: "apply_latency_ns": 0 2024-04-01T06:14:49.279 INFO:tasks.workunit.client.0.smithi067.stdout: }, 2024-04-01T06:14:49.279 INFO:tasks.workunit.client.0.smithi067.stdout: "alerts": [] 2024-04-01T06:14:49.279 INFO:tasks.workunit.client.0.smithi067.stdout:} 2024-04-01T06:14:49.280 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-04-01T06:14:49.280 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-04-01T06:14:49.545 INFO:tasks.workunit.client.0.smithi067.stderr:dumped all 2024-04-01T06:14:49.557 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-01T06:14:49.558 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-01T06:14:49.666 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-01T06:14:49.667 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-01T06:14:49.667 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-01T06:14:49.668 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-01T06:14:49.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-01T06:14:49.669 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-01T06:14:49.669 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-01T06:14:49.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:14:49.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-01T06:14:49.670 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-01T06:14:49.671 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T06:14:49.671 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:14:49.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-04-01T06:14:49.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-04-01T06:14:49.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-04-01T06:14:49.673 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-04-01T06:14:49.674 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-04-01T06:14:49.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-01T06:14:49.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-01T06:14:49.684 INFO:tasks.workunit.client.0.smithi067.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70255 2024-04-01T06:14:49.684 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70255 2024-04-01T06:14:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T06:14:49.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-01T06:14:49.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-01T06:14:49.686 INFO:tasks.workunit.client.0.smithi067.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-04-01T06:14:49.686 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-04-01T06:14:49.686 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-01T06:14:50.028 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-04-01T06:14:50.250 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=55ac34a23078449ebea6c718c2ae4d0b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-04-01T06:14:50.253 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-01T06:14:50.262 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-01T06:14:50.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T06:14:50.319 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-01T06:14:50.329 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-01T06:14:50.329 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T06:14:50.349 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-04-01T06:14:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T06:14:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-04-01T06:14:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 93 +328us[ +328us] +/- 82ms 2024-04-01T06:14:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 9 377 465 -307us[ -308us] +/- 64ms 2024-04-01T06:14:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 +273us[ +273us] +/- 90ms 2024-04-01T06:14:50.353 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T06:14:50.354 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-01T06:14:50.362 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-01T06:14:50.362 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-01T06:14:50.409 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T06:14:50.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T06:14:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T06:14:50.501 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-01T06:14:50.511 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-01T06:14:50.543 INFO:teuthology.task.internal:Duration was 5599.532845 seconds 2024-04-01T06:14:50.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-01T06:14:50.597 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-01T06:14:50.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T06:14:50.640 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T06:14:51.104 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-01T06:14:51.104 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-04-01T06:14:51.104 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T06:14:51.163 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-01T06:14:51.164 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T06:14:51.233 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-01T06:14:51.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T06:14:51.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-01T06:14:51.380 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-01T06:14:51.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T06:14:51.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-01T06:14:51.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:14:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-04-01T06:14:51.492 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T06:14:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T06:14:51.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-01T06:14:51.559 INFO:teuthology.task.internal:Transferring archived files... 2024-04-01T06:14:51.561 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi/7632907/remote/smithi067 2024-04-01T06:14:51.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T06:14:52.739 INFO:teuthology.task.internal:Removing archive directory... 2024-04-01T06:14:52.739 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T06:14:52.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-01T06:14:52.780 INFO:teuthology.task.internal:Not uploading archives. 2024-04-01T06:14:52.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-01T06:14:52.810 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-01T06:14:52.810 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T06:14:52.826 INFO:teuthology.orchestra.run.smithi067.stdout: 265718 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 1 06:14 /home/ubuntu/cephtest 2024-04-01T06:14:52.826 INFO:teuthology.orchestra.run.smithi067.stdout: 403331 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 1 04:48 /home/ubuntu/cephtest/mnt.0 2024-04-01T06:14:52.826 INFO:teuthology.orchestra.run.smithi067.stdout: 403332 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 1 04:59 /home/ubuntu/cephtest/mnt.0/client.0 2024-04-01T06:14:52.826 INFO:teuthology.orchestra.run.smithi067.stdout: 403374 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 1 04:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-01T06:14:52.827 INFO:teuthology.orchestra.run.smithi067.stdout: 415335 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 06:14 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-04-01T06:14:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T06:14:52.828 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-01T06:14:52.828 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-01T06:14:52.833 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-01T06:14:52.861 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-01T06:14:52.937 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi067 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-01T06:14:52.937 INFO:teuthology.nuke:Checking targets against current locks 2024-04-01T06:14:52.962 DEBUG:teuthology.nuke:shortname: smithi067 2024-04-01T06:14:52.963 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T06:14:52.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:00:02-rados-squid-distro-default-smithi/7632907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 04:33:16.457646', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDVqi0A7C6HAQGxP/GypzQQl4gTlLvjVme2mh56L5LfdYSfdgC5RR/+RGzME5WAzu9ZQnaA3Xr/nrpkBwgJBuws='} 2024-04-01T06:14:53.039 INFO:teuthology.orchestra.console.smithi067:Power off 2024-04-01T06:14:53.040 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T06:14:53.063 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-04-01T06:14:53.064 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T06:14:53.092 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-01T06:14:57.094 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T06:14:57.204 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-01T06:15:01.209 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T06:15:01.320 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-04-01T06:15:05.323 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T06:15:05.436 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-04-01T06:15:05.537 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-04-01T06:15:05.646 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 5599.532844543457 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi067 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384f31f3d3aee7f8ba3f60846b8844c049165583 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=55ac34a23078449ebea6c718c2ae4d0b status: fail success: false 2024-04-01T06:15:05.647 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T06:15:05.741 INFO:teuthology.run:FAIL